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Abstract. We give a survey of several models of irreducible complementary series repre- 
\^ ", sentations and their limits, special representations, for the groups SU (n, 1) and SO{n, 1), 

' including new ones. These groups, whose geometrical meaning is well known, exhaust the 

■ list of simple Lie groups for which the identity representation is not isolated in the space 

of irreducible unitary representations (i.e., which do not have the Kazhdan property) and 
CJ ' hence there exist irreducible unitary representations of these groups — so-called "special 

. representations" — for which the first cohomology of the group with coefficients in these 

\^ \ representations is nontrivial. By technical reasons, it is more convenient to consider the 

groups 0(n, 1) and U{n, 1). Most part of the paper is devoted to the group U{n, 1). 
The main emphasis is on the so-called commutative models of special and complemen- 
I tary series representations: in these models, the maximal unipotent subgroup is repre- 

sented by multiplicators in the case of 0{n, 1), and by the canonical model of the Heisen- 
berg representations in the case of U (n, 1). Earlier, these models were studied only for the 
group S'L(2,R). They are especially important for realization of nonlocal representations 
of current groups, which will be considered elsewhere. 

We substantially use the "density" of the irreducible representations under study of 
SO{n, 1): their restrictions to the maximal parabolic subgroup P are equivalent irreducible 
representations. Conversely, in order to extend an irreducible representation of P to a 
representation of SO{n, 1), we must additionally define only one involution. For the group 
CN ' U{n, 1), the situation is similar but slightly more complicated. 
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Introduction 

0.1. Special representations. An irreducible unitary nonidentity representation Tg of a 
group G in a Hilbert space L is called special if it can be extended to an indecomposable 
reducible representation T in a one-dimensional extension L©{^} of the space L: Tg{h, ^) = 
{Tgh + ^(^r), where the map b : G ^ L satisfies the condition 

K9192) = b{gi) + Tg^b{g2) for any gi,g2^ G}^ 

Such a map is called a 1-cocycle, and the representation is indecomposable if and only if 
this cocycle is nontrivial, i.e., the function h[g) cannot be written in the form h{g) = Tg^ — ^ 
for any ^ G L. Given a representation T of the group G, we define, in the usual way, the 
additive first cohomology group H^{G,T) of G with values in this representation: the 
quotient of the additive group of cocycles by the subgroup of trivial (i.e., cohomologous to 
zero) cocycles. Thus the fact that an irreducible representation T is special means that the 
first cohomology with values in this representation is nontrivial. 

Cocycles with values in the identity representation of a group are additive one- 
dimensional characters of this group; hence for groups G having such nontrivial characters, 
it is natural to consider that the identity representation is also special. In this case, the co- 
homology group of the identity representation is precisely the group of additive characters 
of G. 

In [1], it is proved that every special nonidentity representation of a locally compact 
compactly generated group cannot be separated (in the Fell-Jacobson topology) from the 
identity representation; in other words, every open neighborhood of a special representation 
has a nonempty intersection with every open neighborhood of the identity representation. 
Thus for groups having a nonidentity special representation, the identity representation 
is not isolated in the set of all unitary irreducible representations; in other words, such 
groups do not have the Kazhdan property (T). Starting from the pioneering paper [3] and 
subsequent presentations (see, e.g., [4]), many studies have been devoted to groups with 
the Kazhdan property (one of the recent books is [5]); but we are interested in semisimple 
groups that do not have this property. In [1], the set of nonidentity irreducible unitary 
representations that cannot be separated from the identity representation was called the 
core, and these representations themselves, by a clear reason, were called infinitesimal 
representations. As one can easily check on the example of the group of Euclidean motions, 
not every infinitesimal representation is special. 



^)Note that the term "special representation" in [49] has another meaning. 
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It may happen that a special representation belongs to the closure of the identity rep- 
resentation. This is equivalent to the fact that a special representation contains almost 
invariant vectors e > ^ gi, g2, ■ ■ ■ Qk ^ G3h G L, \\Tg^h — h\\ < e). 

In [1], it was also conjectured that every locally compact group without property (T) 
has a special representation (possibly, the identity one). This was proved by Y. Shalom 
[7] (see also [5] and references therein), even in a stronger form: every such group has a 
special representation, and, moreover, not only the ordinary cohomology group with values 
in this representation is nontrivial, but the strict cohomology group is also nontrivial. (The 
group of strict cohomologies is the quotient of the group of cocycles by the closure of the 
subgroup of trivial cocycles, see [5]). However, the problem of computing the cohomology 
and explicitly describing special representations, and even that of finding the number of 
such representations, is far from being solved. It is known that for Abelian and nilpotent 
groups, only the identity representation is special. Even for countable solvable groups, the 
situation is more complicated. The cohomology group was studied in a number of papers, 
see, e.g., [8] and the books [27, 29]; for the cohomology of the group of automorphisms of 
a tree, see [2]. 

The squared norm of a 1-cocycle is a conditionally positive definite function on the 
group. For semisimple groups of rank 1, such functions were described in [12] and, slightly 
later and independently, in [9]. In this case, the fact that a cocycle is not cohomologous 
to zero if and only if its norm, regarded as a function on the group, is unbounded, follows 
from the results of [12]. In the general form, for arbitrary unitary representations, it was 
apparently first proved in [6] (see also [5]). 

Among the simple Lie groups, only SO{n, 1) and SU{n, 1) have special representations, 
see [12]. The groups SO{n, 1), n > 2, have exactly one special representation, and the 
groups SU{n, 1) and 5*0(2, 1) have exactly two special representations [29, pp. 168-169]. 
They are the subject of this paper. As to groups of rank higher than 1 and the groups 
Sp{n, 1), for which the identity representation is isolated in the space of irreducible unitary 
representations, similar opportunities appear for them provided that we consider nonuni- 
tary representations with invariant bilinear, but not positive definite, form; in this wider 
class, special representations exist for every simple Lie group, but it seems that this op- 
portunity is not yet investigated in detail. 

Though for groups of rank 1, special representations as such were of course known before 
the paper [12], only that paper made clear their link to cohomology, as well as to unbounded 
conditionally positive definite functions and the canonical state from [10]; see comments 
on these two papers in [13, 11]. 

One of the important applications of nonidentity special representations is as follows: 
using such a representation, one can construct an irreducible nonlocal unitary representa- 
tion of the corresponding current group G^, i.e., the group of bounded measurable maps 
X G with pointwise multiplication, where X is a measurable space; this question is 
considered in the series of papers [10, 12, 14, 15, 16, 17, 18, 19], [20, 21, 33]. However, we 
do not consider these applications in this paper. 



2)For representations of infinite-dimensional groups, see also [39, 40, 4f, 42, 43, 44, 45], [22, 23, 24, 25], 
and the monographs [27, 28]. See also the monographs [29, 30] and the papers [34, 26]. 
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This paper is devoted to irreducible unitary representations of the groups 0(n, 1) and 
U{n, 1), i.e., the groups of hnear transformations in M"'"'"''" and C""*"^ preserving, respec- 
tively, the quadratic and Hermitian forms of signature (n, 1). Note that, up to central 
subgroups, they are the groups of motions of the real and complex Lobachevsky spaces 
0(n, l)/0(n) X 0(1) and U{n, l)/U{n) x [/(l).^) 

We study only the special representations of the groups 0(n, 1) and U{n,l) and the 
closely related complementary series representations, leaving aside the general representa- 
tion theory of these groups, which is presented in an extensive literature. We place emphasis 
on the properties that distinguish these representations and their realizations from other 
representations: in particular, 

the restriction of a special representation of the group 0{n, 1) or U{n, 1) to the maximal 

parabolic subgroup P remains a special irreducible representation. 
Thus the study of the special representations of 0(n, 1) and 1) reduces to studying 
the special representations of their maximal parabolic subgroups. 

The structures of the complementary series representations and the special representa- 
tions in the case of f/(n, 1) are richer and more beautiful than the analogous structures in 
the case of 0(n, 1). We consider these groups separately; the case of 0(n, 1) is presented 
more briefly, and the main part of the paper concerns the group U{n, 1). For representations 
of the group 0(n, 1), see also [18, 19]. 

The theorems presented in the paper describe various models of the representations 
under study, including new ones. 

0.2. The special representations of the groups 0(n, 1) and ?7(n, 1) as limits of the 
complementary series representations. The special representations of 0(n, 1) and 
C/(n, 1) can be obtained by passing to the limit from the so-called complementary series 

representations, which are of independent interest. 

Let G be one of these groups and P be its maximal parabolic subgroup (which is unique 
up to conjugation). Complementary series representations of the group G are its irreducible 
unitary representations induced from one-dimensional nonunitary representations of the 
subgroup P. These representations depend on a real parameter A, which lies in the interval 
(0,n— 1) in the case of 0(n, 1), and in the interval (0,2n) in the case of C/(n, 1). Their 
description in the case of G = SL{2,K) can be found in [31] for if = E and C, and in 
[32] for a non- Archimedean field K. 

Complementary series representations can be realized, for example, in Hilbert spaces 
of functions on the sphere S = G/P. Other realizations are described below. 

At the endpoints of the interval (0, n—1) in the case of 0(n, 1) and, respectively, (0, 2n) 
in the case of U{n, 1), the space becomes reducible. 

In the case of 0(n, 1), for A = the norm in the space degenerates on a subspace 
L of codimension 1. The special representation T of 0(n, 1) is realized in this subspace L 
with the norm 



^^Replacing the groups SO{n, 1) and SU (n, 1) by their extensions 0(n, 1) and U (n, 1), while not causing 
any substantial changes, simplifies constructions. 



5 



The associated nontrivial 1-cocycle is given by the formula b{g) = Tg^ — ^, where ^ ^ L. 

For A = n— 1, the norm ||/||a degenerates on a one-dimensional subspace Lq. The special 
representation is realized in the quotient by Lq with the norm 



= lim 

The special irreducible unitary representations of the group 0(n, 1) corresponding to the 
endpoints of the interval (0,n— 1) are equivalent. 

In the case of C/(n, 1), for A = the norm ||/||a also degenerates on a subspace L of 
codimension 1. In contrast to the case of 0(n, 1), the norm ||/|| = ^im^ ^^ix^ in L forn > 1 

degenerates on some infinite-dimensional subspace Li C L. The Hilbert space L/Li (with 
the inherited norm) splits into the direct sum of two irreducible nonequivalent invariant 
subspaces and H-. The representations of the group C/(n, 1) in the subspaces H-^. and 
H- are special representations. 

For \ — 2n with n > 1, the norm ||/||a degenerates on a subspace L of infinite codi- 
mension. The norm ||/|| = lim ^^^J^^ on L degenerates on a one-dimensional subspace 

A — > 2n 

Lq C L. The representation of the group U{n, 1) in the Hilbert space L/Lq splits into the 
direct sum of two nonequivalent special irreducible representations. 

The special unitary representations of U{n, 1) corresponding to the endpoints of the 
interval (0, 2n) are equivalent. 

0. 3. Matrix realizations of the groups 0(n, 1), U{n, 1) and of some subgroups of 
these groups. Matrix realizations of the groups 0(n, 1) and ?7(n, 1) are determined by 
the choice of a quadratic (respectively, Hermitian) form of signature (n, 1) in the space 
]^n+i (respectively, C""^^). This choice is in turn equivalent to fixing a Cartan subgroup 
of the corresponding group. 

We realize 0(n, 1) and U{n, 1) as the groups of linear transformations in M""*"^ and C 
preserving, respectively, the quadratic form 2xix„+i + ^2 + . . . + and the Hermitian 

form XiXn+l + Xn+lXi + \X2\^ + . . . + \Xn\^- 

In this realization, elements of the groups can be written as block matrices g = \\gij\\ of 
order 3, where the diagonal blocks gu, g22, and gss are square matrices of orders 1, n— 1, 
and 1, respectively. The maximal unipotent subgroups Z C 0(n, 1) and H C U{n, 1) are, 
respectively, the groups of matrices of the form 

/ 1 0\ / 1 0\ 

z= -7* e ,7€E"-\ and h=\ -z* e o] , t eR, z e 
V-^ 7 1/ V*-* - 1/ 

where e is the unit matrix of order n— 1, 77* = + . . . + 7^_i, zz* — |2i|^ -|- . . . -|- 

It is convenient to write elements of H as pairs {t,z), t € E, z € C"^""^, with the 
multiplication law 

{ti,zi) (^2,^2) = (*i +*2 -Imzi^l,^! + Z2). 

The normalizer of these subgroups is the subgroup P of lower block triangular matrices, 

1. e., the maximal parabolic subgroup. It is essential that each of the groups 0(n, 1) and 
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U{n, 1) is algebraically generated by the elements of P and one element 

/O 1\ 
s = e . 
\l oj 

Thus representations of the groups 0(n, 1) and U{n,l) are completely determined by the 
operators corresponding to the elements of the subgroup P and this element s. 

The group P is the semidirect product P = Z\D and P = H \ D, respectively, where 
D is the subgroup of block-diagonal matrices of the form 

d = diag(e~'^, w, e), e^E*, uEO{n—l), in the case of 0(n, 1). 

d = diag(e~^, M, e), e€C*, u G f/(n— 1), in the case of U{n,l). 

The subgroup D can be written as the direct product D = Dq x Di, where Dq is the 
maximal compact subgroup of D. In the case of U{n, 1), Dq is the subgroup of block- 
diagonal matrices d — diag(e,u, e), |e| — 1, u G U{n—1), and Di = is the subgroup of 
block- diagonal matrices of the form d — diag(r~^, e, r), r > 0. In the case of 0(n, 1), the 
subgroups Do and Di are defined in a similar way. 

By Pq we denote a subgroup of P of codimension 1, namely, Pq = Z \ Dq and Pq = 
H X Dq, respectively. 

0.4. The difference in the structures of complementary series representations 
and special representations of the groups 0(n, 1) and f/(n, 1). The difference in the 
structures of representations of the groups 0(n, 1) and ?7(n, 1) is predetermined by the 
structure of their maximal unipotent subgroups Z and respectively. 

The subgroup Z C 0(n, 1) is commutative and isomorphic to the additive group R""^; 
thus all irreducible representations of Z are one-dimensional. Hence in the case of 0(n, 1) 
there exists a model of representations in which the operators corresponding to the unipo- 
tent subgroup act as multiplicators. We call it the commutative model associated with the 
unipotent subgroup. 

The subgroup H C t/(n, 1) is isomorphic to the Heisenberg group of dimension 2n— 1, 
so that it has a one-parameter family of infinite-dimensional irreducible unitary represen- 
tations. In this case, the analog of the commutative model for 0(n, 1) is a direct integral 
of multiples of irreducible infinite-dimensional subspaces of the Heisenberg group. Thus 
the theory of complementary series representations and special representations of U{n, 1) 
merges with the representation theory of the Heisenberg group. 

Note that the Heisenberg subgroup H C U{n, 1), in contrast to the maximal unipotent 
subgroup of 0(n, 1), has the following remarkable property. Every infinite-dimensional irre- 
ducible unitary representation of H can be (uniquely) extended to a unitary representation 
of the subgroup Pq d P oi codimension 1: P = Pq\Di, Di 

It is known that the groups 0(n, 1) and [/(n, 1) are algebraically generated by the el- 
ements of the subgroup P and one element s of order 2. Thus if the space of a unitary 
representation of f/(n, 1) is decomposed into a direct integral of infinite-dimensional sub- 
spaces invariant and irreducible with respect to the Heisenberg subgroup i7, then, in order 
to describe this representation, it suffices to define only the action of the operators of the 
one-parameter subgroup Di and the operator corresponding to the element s. 
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0.5. Restrictions of representations of 0(n, 1) and U{n, 1) to the maximal com- 
pact and parabolic subgroups. The difference between 0(n, 1) and U{n, 1) shows itself 
also in the restrictions of complementary series representations and special representations 
to the maximal compact subgroup K and the maximal parabolic subgroup P. 

In both cases, the restrictions of complementary series representations to K are pairwise 
equivalent and contain the identity representation with multiplicity one; the spectrum of 
these restrictions is simple only in the case of 0(n, 1). When passing to the special represen- 
tations, in the case of 0(n, 1), the identity representation vanishes; in the case of U{n, 1), 
n > 1, besides the identity representation, a certain infinite-dimensional representation of 
K also vanishes. Each of the remaining irreducible representations of K occurs in one of 
the two special representations with multiplicity one. 

In both cases, the restrictions of all complementary series representations to P are equiv- 
alent to one and the same representation of P induced from the identity representation of 
its block-diagonal subgroup. 

In the case of 0(n, 1), n > 2, this representation of P is irreducible and equivalent to the 
restriction to P of the special representation. Thus on the subgroup P these representations 
coincide. On the whole group 0(n, 1) they differ by the action of the single operator T^. 

In the case of U{n, 1), this representation of P decomposes into a direct sum of pairwise 
nonequivalent irreducible representations. The restrictions of the two special representa- 
tions to P are irreducible and occur in this decomposition as two direct summands. 

Observe the difference between the restrictions of complementary series representations 
and special representations of the group ?7(n, 1) to the Heisenberg subgroup H. 

The decomposition of every complementary series representation contains every infinite- 
dimensional representation of the Heisenberg group with infinite multiplicity. 

The sum of the two special representations ofU{n, 1) decomposes into the direct integral 
of all pairwise nonequivalent representations of the Heisenberg group. 

In the remaining part of the Introduction we briefly describe the main results of the 
paper. 

0.6. Realization of the complementary series representations and the special 
representation for the group 0(n, 1). In § 2 we describe six models of the comple- 
mentary series representations and six models of the special representation of the group 
0(n, 1); these models differ by the choice of coordinate system, functional space, and the 
endpoints of the interval corresponding to the complementary series. 

More exactly, the invariant bilinear functional that determines the Hilbert space of the 
complementary series representation can be defined for any value of the parameter 
A G [0,n— 1]. The values A and (n— 1)— A (that are distinct from the endpoints of the 
interval) give rise to equivalent complementary series representations and J"""^"^; 
their realizations are different. 

This difference in the realization of T'^ and T'^~^~^ is especially vivid when one com- 
pares the cases A = and A = n— 1, when these representations become reducible and 
nonequivalent. Starting from T^ and T'^~^, we can obtain two models of the unique spe- 
cial irreducible representation of the group 0(n, 1). These models are different. Namely, as 
observed above, for A = the special representation is realized in an invariant subspace 
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of CO dimension 1, while for A = n— 1 it is realized in the quotient by a one- dimensional 
invariant subspace. 

Realization of representations also depends on the choice of coordinate system on the 
sphere 5*""^ C R"". Model A uses spherical coordinates, while model B uses the local 
coordinates of the punctured sphere, which determine a bijection between M"^"^ and the 
unipotent subgroup Z. 

Thus we obtain four models for each complementary series representation and four mod- 
els for the special representation of the group 0(n, 1). Two of these four models of the 
special representation are obtained for A = 0, and the other two models, for A = n— 1. 

Besides, one can pass to the Fourier transform in the space of model B realized in the 
local coordinates. Then the operators corresponding to the unipotent subgroup Z will act as 
multiplicators, and we obtain two more models of representations of 0(n, 1) (models C, or 
commutative models), which are especially important in applications to the representation 
theory of the current group 0(n, 1)^ [18, 19]. 

In particular, this realization is convenient for obtaining an explicit formula for the 
nontrivial 1-cocycle b : 0(n, 1) — > L° associated with the special representation (see Sec- 
tion 2.7). 

In the commutative model with A = 0, this 1-cocycle is given by the formula 

b{g) = - ^, 

where 

2 

Here Kp{x) is a Bessel function (^ is a vector that does not belong to and is invariant 
under the maximal compact subgroup K of the group 0(n, 1)). In particular, 

For each model of the complementary series representations, we explicitly describe the 
spherical function and the vacuum vector ^, i.e., a vector from the representation space 
that is invariant under the maximal commutative subgroup. We also describe the em- 
bedding of the spaces of complementary series representations into their tensor product. 
This embedding leads to one of the constructions of representations of the current group 
0(n,l)^. 

0.7. Construction of an irreducible representation of the subgroup P C U{n, 1). 
In §§ 3-6 we consider representations of the group U{n, 1) and its parabolic subgroup P. 

We begin in § 3 with a construction of a class of unitary representations of the group P 
based on the construction of irreducible representations of the Heisenberg group H. It is 
known that every infinite-dimensional unitary representation of H is determined by a real 
parameter p 7^ (the Planck constant). We consider the Bargmann realization of these 
representations. In this realization, the representation with parameter p > is defined in 
the Hilbert space H{p) of entire analytic functions f{z) on C"""^ with the norm 

f 1/(^)1' e-HN'dM(z), (0.1) 
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where = = + . . . + l-^n-iP and dfj,{z) is the Lebesgue measure on C""-^. The 
operators corresponding to elements h = {to, zq) ^ H have the form 

Thf{z) = 6^(^*0-5 l^op-^^S) f{z + 20). (0.2) 

Analogously, the representation with parameter p < is defined in the Hilbert space 
^{{p) of entire antianalytic functions on with the norm (0.1). The operators corre- 

sponding to elements h — (to, zq) E H have the form 

Thf{z) = e^^*°-l^l l^ol'+^o^*) f{z + zo). (0.3) 

Each of these representations can be extended to a unitary representation of the group 
Pq = H \ Do- Namely, the operators Td corresponding to elements d — (e, w, e) € Dq are 
given, for every p, by the formulas 

Tdf{z) = fiezu). (0.4) 

To construct unitary representations of the group P, we associate with every A ^ direct 
integrals of the spaces H{p): 

roo rO 

nX= n{p)p^-^dp and n^= I n{p)\p\^-^dp. 

Jo J-00 

The representations of the group Pq in these spaces can be extended to unitary represen- 
tations of the group P = PqX Di. Namely, the operators T^, corresponding to elements 
d — diag(r~'^, e, r) € Di have the form 

Tdf{p,z)=f{r^p,r-'z)r\ (0.5) 

The representations of the group P in the spaces (respectively, 7Y^) defined in this 
way are irreducible and pairwise equivalent; the representations in and are not 
equivalent. 

In § 6 it is shown that these representations of the group P can be extended to special 
representations of the group U{n, 1). 

0.8. Representations of the group P in the spaces H^. In §4 we construct a family 

of reducible unitary representations of the subgroup P C U{n, 1) in spaces and H^. 

As shown later in § 5, the representations of the subgroup P in the spaces 

can be extended to (irreducible) complementary series representations of the whole group 

U{n,l). 

The spaces are defined as the direct integrals 

poo rO 

I H{p)p^-^dp and H^^ H{p) \p\^-^ dp, 
Jo J-00 

where H{p) is the Hilbert space of all functions f{z) on C"^~^ with the norm (0.1). 

In H{p), a unitary representation of the group Pq acts; it is given by (0.2) and (0.4) for 
p>0, and by (0.3) and (0.4) for p < 0. 
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The space H{p) decomposes into a direct sum of invariant Po-irreducible pairwise 
nonequivalent subspaces: 

oo 
m=0 

where Hm{p) with p > and p < is cychcahy generated by the vectors "z^ = . . . "z^^i 

and = z^^ . . . z^^Si , respectively, where ki + . . . + = m. In particular, Hq{p) — H{p). 

The representation of the Heisenberg subgroup in each subspace Hm{p) is a multiple of 
the irreducible representation with parameter p. The multiplicity is equal to the number 
of partitions of m into the sum of n—1 nonnegative integer summands. 

Each space H{p) has an orthogonal basis {fpq | p, g G Z^~^} compatible with the 
action of the Heisenberg group H. For every fixed p, the vectors fpq, q G form 
an orthogonal basis in the subspace of an irreducible representation of the group H. We 
describe the matrix elements of the representations of the group Pq in the same subspace 
in terms of this basis. 

The representations of the group Pq in the subspaces can be extended, according to 
(0.5), to unitary representations of the group P. These spaces are reducible with respect 
to P and decompose into a direct sum of P-irreducible pairwise nonequivalent subspaces: 

oo 
m=0 

where 

rex rO 

{Hl)m= Hm{p)p^-^dp and {H^)m = Hm{p) \p\^-^ dp. 

Jo J-oo 



0.9. Models of the complementary series representations of the group U{n, 1). 

In § 5, by analogy with the case of the group 0(n, 1), we construct six different models of 
the complementary series representations of the group C/(n, 1). We start from the known 
realization of these representations in a space of functions on the unitary unit sphere 
S C C"' (model A), and then pass to their realization in a space of functions on the 
Heisenberg group i.e., in the space of functions (p{t, 0) on M x C"~^ (model B). 

In the next model (model C), the complementary series representations of the group 
U{n, 1) are realized as direct integrals of multiples of irreducible representations of the 
Heisenberg group. They are obtained by passing from functions (^(t, z) in model B to 
functions ip{p, z) on E X C""\ which are given by the integral 

/+00 
e-'P^ip{t, z) dt. 
-00 

In this model, the space of the complementary series representation is a direct integral 
of Po-invariant subspaces L{p): 

/+00 
L{p) H^-^dp, 
-00 
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where L{p) is the space of functions 'il){z) on ^ with the norm 

llV^llp = \p\^''~'^ I R\{P, zi, Z2) ijjip, z) ijjip, z) dp{zi) dp{z2). 

An exphcit expression for R\ in terms of the Bessel function Ki-x (t) is given in the main 

2 

text. 

The operators of the subgroup Pq in the space L{p) are given by the same formulas as 
for the subspace H{p) defined in Section 0.8. 

The total number of models of each complementary series representation of the group 
C/(n, 1) is equal to 3 rather than 6, because the representations in the spaces and L?''^~^ 
are equivalent. 

Let us formulate the main result of §5. It is proved that the representations of the 
subgroup Pq in the spaces L{p) and H{p) are equivalent. Thus there exists an isomorphism 
of Hilbert spaces 

J : H{p) ^ L{p) 

commuting with the operators of the group Pq. Since H{p) = 0^=o^to(^)' where H^ip) 
are Po-invariant irreducible pairwise nonequivalent subspaces, it follows that on each of 
them the operator J is a multiple of the identity operator: J = id on Hm{p)- (In the 
main text, we explicitly calculate the coefficient Cm-) 

The isomorphism J : H{p) ^ L{p) induces an isomorphism of Hilbert spaces 

commuting with the action of the whole group P (and not only the subgroup Pq C P). 

In view of the isomorphism — > L^, the action of the group U{n, 1) can be transferred 
from the space to the space H^. 

Thus the space can be regarded as a new model of the space of a complementary 

series representation. 

In this new model, the operators of the subgroup P are given by the same formulas as in 
the space L^. Hence, to describe a representation of the group U{n, 1) in the new model, 
it suffices to know only the operator Tg. 

0.10. Models of the special representations of the group U{n, 1). In § 6 we construct 
six models of the special representations of the group U{n, 1). They are obtained from the 
models of the complementary series representations by passing to the limit as A ^ and 
A 2n. 

The structure of the spaces and L^", which are the limits of the spaces of com- 
plementary series representations, was already discussed in Section 0.2. 
Let us describe the main model (model C, A = 0). 

The special representations of the group t/(n, 1) are realized in the spaces 
^+ = / ^P)p~^dp, ^- ^ ^P)\p\~^dp. 

Jo J-00 

Here C{p) is the quotient of the space of functions ifj{z) on C""-^ with the norm 

= |p|2«-2 / el^l(«^(^^'^2)-|^^l'-|^2l')V'(^i)VK^^^/^(^i)^^M^2), 
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where 

{Z1Z2 for p > 0, 
(0.6) 
Z2zl for p<0, 

by the subspace of functions of norm 0. 

The spaces jC{p) are irreducible and pairwise nonequivalent under the action of the 
Heisenberg group H. Thus the special representations of the group U{n, 1) decompose into 
direct integrals of irreducible pairwise nonequivalent representations of the Heisenberg 
group. 

There is another realization of the special representations of U{n, 1). Namely, it is proved 
that there exists an isomorphism of Hilbert spaces 

commuting with the action of the subgroup P, where Tt^ are the spaces defined in § 0.7 of 
the Introduction. 

In view of this isomorphism, the action of the group U{n,l) can be transferred from the 

spaces C± to the spaces H±. 
We call the realizations of the special representations of U{n, 1) in H± the Bargmann 
models. In these models, the action of the operators of the subgroup P is defined from the 
beginning, so that in order to give a complete description of a representation of U{n, 1), it 
suffices to determine only the action of the operator Tg. 

It is establish that in the case of the action of T5 on functions of the form 

where z'^ — z'l^ . . . z^^^^, \k\ — ki + . . . + kn-i, and Rea > 0, is given by 

T,(pl^l e-^P z^) = a-l^l pl^l e-a z^. 

In particular, Ts{e~^P) = e~a. In a similar way Tg is defined on the space 

In the Bargmann model, the nontrivial 1-cocycles h± : U (n, 1) 71%. are given by 

b±{g)=Tg^±-^±: e± = X±(p)e-"^", 

where x± ^^re the characteristic functions of the half-lines p > and p < 0, respectively. 
Since h±{g) = on the maximal compact subgroup, this function is uniquely determined 
by its restriction to the subgroup Pi, a one- dimensional extension of the Heisenberg group 
H. 

0.11. Canonical representations. In the construction of the special representations of 
the groups 0(n, 1) and [/(n, 1), one can replace the complementary series with another 
one-parameter series of unitary representations, the canonical representations. 

Canonical representations were first introduced in [10, 17] in connection with construct- 
ing representations of the current groups 0(n, 1)^ and U{n^ 1)^. 

In Appendix 1, we first give a general definition of canonical representations for an arbi- 
trary locally compact group G, and then explicitly construct the canonical representations 
of the groups 0(n, 1) and f/(n, 1) and establish their relation to the complementary series 
representations and the special representations. 

In Appendix 2, we derive some formulas used in the paper. 
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§ 1. The groups 0(n, 1), U{n, 1) and some subgroups of these groups 

1.1. The groups 0{n, 1) and U{n, 1). By 0{n, 1) and U{n, 1) we denote the groups of 
all linear transformations in E"'"'"^ and C"'"'"^ preserving, respectively, the quadratic and 
Hermitian forms of signature (n, 1); this condition uniquely determines the groups 0(n, 1) 
and U{n, 1) up to isomorphism. In what follows, we use two matrix models of these groups, 
which are determined by the choice of the corresponding quadratic and Hermitian form. 

1.1.1. Model a. The quadratic and Hermitian forms are, respectively, 

2 I I 2 _ 2 

Xi -\- . . . -\- X^j^l, 

In this model, elements of the groups are written as block matrices 

fa P\ 

where the diagonal blocks a, S are matrices of orders n and 1, respectively. In this notation, 
0(n, 1) and U{n, 1) are, respectively, the groups of real and complex matrices g of order 
n + 1 satisfying the condition 

gag* = a, where cr ^ > (1-1) 

i.e., 

aa* - 1313* = e, a-^* - (35* = 0, SS* - 77* = 1. 

Here e is the unit matrix of order n and * denotes conjugation, i.e., a* = in the real 
case, and a* = in the complex case. 

1.1.2. Model b. The quadratic and Hermitian forms are, respectively, 

2 2 
'^XiX'fi-^-i -\- X2 "h • • • "h ^77,7 

XiXn+l + Xn+lXi + |x2p + . . . + \Xn\'^ ■ 

In this model, elements of the groups are written as block matrices 

^gn gi2 gi'd' 
g = \ g2i g22 ^23 

^^31 5'32 5'33^ 

where the diagonal blocks are matrices of orders 1, n— 1, and 1, respectively. In this no- 
tation, 0(n, 1) and U{n, 1) are, respectively, the groups of real and complex matrices g of 
order n + 1 satisfying the condition 




gsg* = s, s = I e I , (1.2) 
where e is the unit matrix of order n— 1. 
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Remark. Model a is convenient for describing the maximal compact subgroup in U{n, 1). 

Namely, the latter subgroup in model a consists of block-diagonal matrices 9 — (]q ; 

where u e 0(n), e = ±1 in the case of 0(n, 1); and u e U{n), e e C * in the case of U{n, 1). 
Model b, mostly used in this paper, is convenient for describing the unipotent subgroup. 

1.1.3. Transition from model a to model b and vice versa. Note that the matrices a and s 
are related by 

1 / 1 -1 

a ^sa — a, where a — — ^ e 

V 2 y 1 1 

It follows that the transition from model a to model b reduces to the transformation 
g aga~^. Let us give explicit formulas for this transition. For this, let us write the blocks 

of matrices o = I '^1 in model a in the form 

where a22 is a matrix of order n— 1 and P2, I2 are, respectively, a column and a row of 
length n—1. Then the transition from model a to model b is given by the following formula: 

/i(aii + 5-/3i-7i) \{aii - 5 + f3i - 

fa /3 \ a2i-/32 ^ a2i+l32 

\ 6 } — — V2~ 

^ y(«ii-^-A+7i) ^an+S + Pi+ji)) 

Conversely, the transition from model b to model a is given by 



9ii 912 913 

921 922 923 
.931 932 933j 



.7 



where 



_ [U9n+933 + 9i3+93i) ^_ [U-9n+933 +9i3-93i 

^ — I .921 +323 „ ' ~ I -g21+ff23 

7 = {l^{-9ll + ^33 - 913 + ^31), ), = -[gii + 5f33 - 5fi3 - 5f3i). 

1.2. The subgroups Z and iif. Denote by Z C 0(n, 1) and H C ?7(n, 1) the following 
subgroups of block matrices in model b: 

( 1 



z = 



v 




-7* e 0|,7€E"-\ and h=\ -z* e | , t G M , ^ € C"-\ 

.77* 




2 

where e is the unit matrix of order n—1. They are the maximal unipotent subgroups of 
the corresponding groups. The first of them is isomorphic to the additive group M^-i; the 
second one is the Heisenberg group of dimension 2n— 1. 
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Elements of H will be written as pairs h = {t, z) or h = {(, z), where = it — In 
this notation, the product of elements of H is given by the formulas 

(t, z) [t', z') = {t + t' - lni{zz'*), z + z') 

or 

(c,^) {C,z')^{c + (:'-zz'\z + z'). 



Here and in what follows, zz'* — ziz[ + . . . + Zn-iz'^_i. 

According to the formulas for the transition from one model to the other, in model a 

the subgroup H consists of block matrices h—[^ ^ 



a — 




(1.3) 



1.3. The subgroups D, Dq? and Di. Denote by D the subgroup of block-diagonal ma- 
trices in model h. It consists of all block-diagonal matrices of the form 

d = diag(e~"'^, w, e), e G M*, w € 0(n— 1) in the case of 0(n, 1), 

d = diag(e~^, M, e), eeC*, u^U{n—l) in the case of C/(n, 1). 
The group D decomposes into the direct product 

D = DqxDi, 

where, in the case of ?7(n, 1), Do = U{1) x C7(n— 1) is the subgroup of block- diagonal 
matrices of the form 

d = diag(e, w, e), where |e| = 1, w€f/(n— 1), 
and Di = M ^ is the subgroup of block-diagonal matrices of the form 

d — diag(r~^, e, r), where r > 0. 
In the case of 0(n, 1), the subgroups Do and Di are defined in a similar way. 

1.4. The subgroups P, Pq? and Pi. Denote by P the normalizer of the maximal unipo- 
tent subgroup, i.e., the maximal parabolic subgroup. Obviously, in model h the subgroup P 
coincides with the subgroup of all lower block triangular matrices and thus can be written 
as the semidirect product 

P = H\D. 

Further, denote 

Pq = H\Dq, Pi = H\Di. 

Proposition 1.1. In model h, the groups 0{n, 1) and U{n, 1) are algebraically generated 

1 

by the elements of the subgroup P and the element s = | e 

1 
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Remark. In view of this proposition, in order to describe a representation of the group 
0{n, 1) or U{n, 1), it suffices to determine, apart from its restriction to the subgroup P, 
only the involution operator / corresponding to the element s. 

Part 1. Representations of the group 0(n, 1) 

§ 2. Models of irreducible unitary complementary series representations of 
the group 0(n, 1) and of the special representation of this group 

Every complementary series representation is determined by a real number A from the 
interval < A < n — 1. Let us describe three models of these representations. 

2.1. Model A: realization in a space of functions on the unit sphere 5" C M". The 

unit sphere 

5 = {a;€E"| |a;|2 = + . . . + = 1} 

is a homogeneous space of the group 0(n, 1). Namely, if this group is realized in matrix 
model a, then its action on S is given by the formula 



ujg = (a;/5 + 5) ^ {iva + 7) for 9 ^ 



2 



The complementary series representation with parameter A acts in the complex Hilbert 
space of functions /(a;) on S with the norm 

/ {l-{u;,u;'))-^f{u;)W)ckjdu;', (2.1) 
JSxS 

where (a;, a;') = c^io;^ + . . . + cuncu^ and dtv is the invariant measure on S. 

Remark. Expanding (1 — (a;,a;'))~'^ into a power series, we can also write the norm in the 
following form: 

Here = E^^^i, k\ = Uki\, o;^ = U^t'- 

The operators T^, y = ' ^^^^ representation are given by the formula 

Tg'f{u)^fiu;g)\u;P + S\'--+\ (2.2) 

The group property of these operators is a consequence of the following relation for the 
function 6(0;, g) — |a;/5 + S\: 

b{uj, gig2) = b{uj, gi) b{ujgi, ^2) for any gi,g2e 0{n, 1) 

(the 1-cocycle property). The unitarity follows from the relations 

1 - {uj, uj') = (1 - {ujg, uj'g)) b{u;, g) b{uj' , g) 

and 

d({i)n\ 

b {u>,g) for every G 0(n, 1). 



dw 
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Proposition 2.1. Complementary series representations and of the group 0{n, 1) 
are equivalent if and only if X + fi = n—1 or X = fj,. 

An intertwining operator R'^ : j^n-i-X giuen by the following formula: 



Js 



2.2. Model B: realization in a space of functions on the maximal unipotent 
subgroup Z ~ E*^"^. We construct this model using matrix realization b of the group 
0(n, 1). Let us define an action z ^ zg oi the group 0{n,l) on Z by the formula 

zg = 

where z' E Z is determined from the decomposition zg = b^z' with 6+ being an upper 
block triangular matrix. In the coordinates 7 e on Z and the block coordinates gij 

on 0(n, 1), this action is given by the following formula: 

|7|2 _ U|2 
19 = ( ^913 + 7^23 + ^33) ( ^912 + 7922 + ^32), (2.3) 

where gij are the entries of a block matrix g € 0(n, 1). In particular, 

19 = 1 + 10 for g = ^(70) € Z- 
19 = e~^-fu for g = diag(e"\ u, e); 

|^|2- 

Remark. The transformation 1 ^ ig has the following geometric interpretation. Let Y be 
the manifold of one- dimensional subspaces in M""*"^ lying in the light cone 

2xiXn+i + X2 + . . . + = 0. 

The group 0(n, 1), as a group of linear transformations in M"""*"^, acts transitively on Y. 
We use the right notation for this action: y yg. Note that in another interpretation 
Y is the absolute of the n-dimensional Lobachevsky space realized as the collection of 
one-dimensional subspaces in E "^"'"■^ lying inside the light cone. 

Let us realize Y\yo, where yo = (1, 0, . . . , 0), as the section of the cone by the hyperplane 
Xn+i = 1, i.e., as the set of points in E""*"-^ of the form 

|7|2 

( ^,7i,---,7n-l,l), 

where 7 = (7^, . . . ,7n-i) ^ and I7I = (X]7i^)"^^^- According to this realization, there 

is a natural bijection Y \yo ^ M"^"^, so that the action of the group G on y induces its 
action 7 ^ 7^ on M"'"-^. We emphasize that this action is not linear. 

Further, we define a function /5(7, ^r) by the formula 

I7P 

/5(7,y) = 1-^^13 + 7^23+^331, 7eM"-\ g^G. (2.4) 

In particular. 
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/5(7, ^/) = 1 for 5f € Z; 

(^{1^9) = |e I for 5f = diag(e"\u,e); 



\1 



2 



2 

It follows from the definition that /3(7, is a 1-cocycle of the group 0(n, 1) with values 
in M*, i.e., 

^ (3{-f,gi)P{^gi,g2) for any 7GM''"^ and 91,92 eG. (2.5) 

In model B, the complementary series representation with parameter A acts in the 
complex Hilbert space of functions on Z = M"^"^ with the scalar product 

(/i,/2) = / i7-7'r''/i(7)Myorfyrf7", (2.6) 



,n-lxRn-l 



where d^y — dji . . . d'jn-i is the Lebesgue measure on E"' ^. The operators of this repre- 
sentation have the form 

?^,'/(7)=/(7^)/3^-"+'(7,^), (2.7) 
where jg and /5(7, fif) are given by (2.3) and (2.4), respectively. In particular, 

7^^/(7) = /(7 + 7o) for z = z{jo)^Z; (2.8) 

rdV(7) = |er""+V(e" V) for d = diag(e-\ u, e); (2.9) 

27 ,/|7|2xi-n+A /O 1 



e I . (2.10) 
VI 

The group property of these operators immediately follows from property (2.5) of the 
function l3{'~f,g); the unitarity follows from the relations 

d{jg)=P^-''{j,g)dj for every ^ € 0(n, 1), (2.11) 

where dj = dji . . . d'jn-i, and 

|7 - 7? = hg - I'g? Pii, g) g) (2.12) 

for any 7,7' e M'^"^ and g e 0{n, 1). 

One can easily check (2.11) and (2.12) for elements from Z, D and for the element s. 
Then it follows from the 1-cocycle property of /3(7, g) that these relations hold for every 
element g G 0(n, 1). 

Proposition 2.2. The functions F{uj) on S and 7(7) on M"'"^ in models A and B are 
related by 

/(a;) = (1 + ^)^+1-- F(r7), (2.13) 
where t'J = uj is given by the formulas 
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2.3. Commutative models of the complementary series representations. Let us 

describe a new model of a complementary series representation T'^ of the group 0(n, 1) in 
which the operators of the subgroup Z act by multiplication by functions. We call it the 
commutative model with respect to the subgroup Z. 

This model is obtained by passing from functions /(7) in model B to their Fourier 
transform 



Theorem 2.1. The commutative model of a complementary series representation is 
realized in the Hilhert space of complex-valued functions on M"^~^ with the norm 

11^11' = '""rlf"" /„_ ICr-"+"l^K)Pde (2.14) 
The operators of this representation are given by the formula 

= / A\^, e', g) ^{O (2.15) 



where 



In particular, 



A\^,C',g)^ [ e^«^'^)-<^''^^))^i-+^(7,^)rf7- (2-16) 



tMO = e-'^^'^'^ ^{0 for z = z{jo)^Z; 
TMO-\^\^^i^Cu) for d = diag(€-\u,e). 

Proof. In the new model, the scalar square is given by the formula 



(2.17) 



■ n-l^Rn-l 



where 



R{^,a= I |7-7'r''e^«^'^)-<^'''^'))d7< = 5(C-0/ 
Equation (2.14) is a consequence of the following relation: 

p 9 -2A -p/ n-1 _ \ \ 

/ |7|-^^e^<«'^) dj = c.f-^y_y. (2.18) 

where the coefficient Cn depends only on n. For a proof of (2.18), see Appendix 2. 

The formulas for the operators of T^ in the new model can be immediately obtained 
from the formulas for these operators in the original model by passing from functions f{'y) 
to their Fourier transforms. □ 
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Proposition 2.3. In the commutative model of the representation T^, the kernel A{^, 

^0 f 

A^(^,^',s) of the operator Tg corresponding to the element s = e has the fol- 



1 0, 



lowing form: 



A{i, ^) = 2^-^ / cos{^x + ^) a;2^-2 dx for n = 2, (2.19) 
Jo X 



fOO 



A{C,e)=Cn'2-^ / r2^-"|r^ + 2r-Vr^^i^(|r^ + 2r-i^|)rfr (2.20) 
for n > 2; here J n-3 is a Bessel function of the second kind. 



-27 

Indeed, since 7s = and /3(7, s) — — — , it follows from (2.16) that 



For n = 2, this immediately implies (2.19). For n > 2, passing to spherical coordinates, we 
obtain 

A(e,0=Cn2-^ / / e^l^^+2^"'^'l^°^'^r2^-" sin"-V^^<P^^r. 
-/q -^o 

Integrating with respect to (p yields (2.20). 

Remark. For n = 2, the kernel A(^, ^') can be expressed in terms of Bessel functions: 

A(e,c') = c(cos(7rA))-i le'rv/' [J2x-i{2'/'m'\) - Ji-2x{2'/'m'\)] for < 0, 

A(^,r) = c (cos(7rA))-i I^TV/' [/2A-i(2^/2|C^1) - Ii-2x{2'/'m)] for > 0. 

2.4. Vacuum vectors and spherical functions. In the space of an arbitrary comple- 
mentary series representation there exists a vector fx invariant with respect to the maximal 
compact subgroup K of the group 0(n, 1), and this vector is unique up to a factor. Let us 
call it the vacuum vector of this representation. 

Proposition 2.4. Up to a factor, the vacuum vector fx G L"^ has the form 

fx{^) — const in model A, (2-21) 

/;^(7) = (1 + ^)A+l-n ^^^g^ ^ (^2.22) 

f\{^ = \^Y^~^^Ky^ n-i (-\/2 I^D) in the commutative model; (2.23) 

here K^{x) is a Bessel function, see [51, Vol. 2]. 

Proof. For model A, the assertion follows from the explicit formulas for the operators of 
the representation T^. By Proposition 2.2, this implies the required assertion for model B. 

The expression for fx in the commutative model can be obtained from (2.22) by the 
Fourier transform. To this end, it suffices to apply the following formula: 

/ (1+ ^)-^e^<^'^) dj = Cr,^-^\^\^-^^-'y' K^_^{a\^\). (2.24) 
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A proof of (2.24) is given in Appendix 2. □ 

By definition, the spherical function of a complementary series representation is the 
function on the group 0(n, 1) given by the formula 

where I is the vacuum vector with norm 1. This function uniquely determines the repre- 
sentation up to equivalence. 

Proposition 2.5. In matrix model a of the group 0{n, 1), the spherical function is given 
by the formula 

M9) = cx\f3\'-^Pl:lm for g=(^^ fj , (2.25) 

where Pif is a Legendre function and \f3\ — (5^ — 1)-^/^ is the norm of a vector (3 G M"'. 

Proof. Since I = const in model a, it follows from the description of the representation 
that 

'^M^cx f \uj(3 + 5\^+^-'' {I- {uj,uj'))-^dwdJ for 
JsxS 

Integrating with respect to co' yields 

M9)=cx [ \uj(3 + S\^+'-''duj. 
Js 

In view of the equation = 5^ — 1, this integral is a function of |5| only. Hence it suffices 
to calculate it for the matrix g G 0(n, 1) with the blocks 

a = ^^^^^ , /3 = (^^^^0^^^ ' 7 = (sinhr,0), 5 = coshr. 
Passing to the spherical coordinates on S, we obtain 

PIT 

ipx{g) = cx / (sinasinhr + coshr)'*^"^"''""^ sin"^~^ o; do;. 
Jo 

The obtained integral can be expressed in terms of the Legendre function Pu{x) according 
to the following formula, see [51, Vol. 1, p. 156(7)]: 

(coshr) = ^|'(^^"^'^) ^ / (sinoisinhr + coshr)^+^(sinc>;)"^'' da. □ 
7rV^r(^ - ii) Jo 

Remark 1. The fact that the spherical functions of two equivalent representations, i.e., 
representations with parameters A and n— 1— A, coincide is a consequence of the following 
relation for the Legendre function: P^j^_]^ (coshr) = Pj^ {coshr). 

Remark 2. The expression for i^xig) in matrix model b of the group 0(n, 1) is obtained by 
replacing \6\ in (2.25) with ^ l^fu + gss - gis - gsil- 
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2.5. Embedding into the tensor product. 

Proposition 2.6. For any real positive numbers Ai, . . . , with ^ A^ < n — 1 there exists 
an isometric embedding 

m 

commuting with the action of the group 0{n, 1). In model B, it is given by the formula 

m 

Tf{^\. . . ,7™) = f{l')X{5{l' - y); (2.26) 

in the commutative model, it is given by the formula 

M^i: ...,U) = ^(6 + + (2.27) 

Proof. Let ( , ) and ( , )to be the scalar products in the spaces L'^ and respectively, 
in model B. Then, obviously, 

» m 



Thus the map r is isometric. It is obvious that it commutes with the action of 0(n, 1). 

The commutative model is obtained by passing from functions / in model B to their 
Fourier transforms. Let v?(^) be the image of f{p() under this transform. Let us check that 
the image of Tf{Wi • • • 1 1^) is ^{^i + . . . + ^rn)- Indeed, according to (2.26), this image is 
equal to 

,. m m 

J fiMll^hi-li)lle'^^-^'^d^i = + □ 



Remark. This embedding leads to one of the models of an irreducible nonlocal unitary 
representation of the current group 0(n, 1)^, i.e., the group of measurable maps X i— > 
0(n, 1), where X is a measurable space. 

2.6. Models of the special representation of the group 0(n, 1), n > 2. The spe- 
cial irreducible unitary representation of the group 0(n, 1), n > 2, is obtained from the 
complementary series representations by passing to the limit as A ^ or A ^ n—1. 

As A ^ 0, the norm ||/||a in the space of a complementary series representation de- 
generates on a subspace of codimension 1. The special representation is realized in the 
subspace with the norm 

dWfW 



lim 



lA 



A^o dX 

For A = n—1, the norm ||/||a degenerates on a one-dimensional invariant subspace Lq of 
the group 0(n, 1). The special representation is realized in the quotient L^~^ /Lq with the 
norm 

" = lim ^ ml 

A— >n— 1 
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The operators of the special representation can be obtained from the operators of comple- 
mentary series representations by passing to the limit as A ^ and A — > n— 1. The unitary 
representations of 0(n, 1) in the spaces and L'^~^ /Lq defined in this way are irreducible 
and equivalent. 

Since each of the spaces and L""! can be defined in three different models, we thus 
obtain six different models of the special representation. 

Let us explicitly describe the spaces and L""^ in models A and B. 

2.6.1. Model A. Here is the subspace of functions f{uj) on the unit sphere S = S'^~^ in 
R satisfying the condition 

/ f{Lo)(ko = 0. 
Js 

The norm in the space L^: 

11/11' = - / \og{l-{iv,cu'))f{cu)7^)dujduj'. (2.28) 
JSxS 

The norm in the space L^~^ of functions on S: 

11/11' = / (l-(a;,a;'))-"+VM7M^rf^'- (2-29) 
JSxS 

The intertwining operator R : ^ L'^~^: 

{Rf){u;) = / log(l - {u;,u;'))f{u;')du;'. (2.30) 
Js 

Remark. In view of (2.30), the norm in can be written as an integral over S: 

ll/f = / {Rf){co)7^duj. 
Js 

2.6.2. Model B. Here is the subspace of functions /(7) on E""-*^ satisfying the condition 

/ f{j)dj = 0. 

The norm in the space L^: 

11/11' = - / log|7-7 1/(7) 7(7)^7 c?7- (2.31) 

The norm in the space L'^~^ of functions on R"""^: 

ll/f = / l7-7r'"+'/(7)7(y)rf7rf7'- (2.32) 

The intertwining operator R : ^ L'^~^: 

11/11' = - / log |7- 71/(70^7'. (2.33) 
Jr"-i 
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2.6.3. Commutative model. 

Theorem 2.2. In the commutative models corresponding to X = and A = n—1, the special 
representation of the group 0{n, 1), n> 2, is realized in the Hilbert spaces of functions f{^) 
on E""-*^ with the norms 

11/11'=/ \fm'\e-''dC and Wff^l 1/(01' 10"-'^^, (2.34) 

respectively. 

Remark. In the exceptional case of the group 0(2, 1), each of the spaces and L'^~^/Lq 
splits into two irreducible subspaces, and we obtain not one but two special representations, 
1) and 2), of this group. In model A, at each endpoint of the interval [0, 1], they are realized 
in the subspaces of functions on the unit circle that are the boundary values of 1) analytic 
and 2) antianalytic functions in the disk \z\ < 1. This case reduces to the case of the group 
f/(l, 1) considered in §6. 

2.7. The nontrivial 1-cocycle associated with the special representation. The 

space L of the special representation has a nontrivial 1-cocycle, i.e., a map b : 0{n, 1) ^ L 
that satisfies the condition 

b{gig2) = ^(^i) + Tg^b{gi) for any gi, g2 G 0(n, 1) 

and cannot be written in the form b{g) = Tg ^ — ^, ^ G L. 

Let us explicitly describe the 1-cocycle b in different realizations of this representation. 

Theorem 2.3. 

o The 1-cocycle b : 0{n, 1) — is given by the formula 

Kg) = Tgi - C, 

where ^ is a vector that does not belong to and is invariant under the maximal 
compact subgroup U of 0{n, 1). The cocycle b is nontrivial. 
o The 1-cocycle b' : 0{n, 1) — > L'^~^ is given by the formula 

b'{g)=Rb{g), 

where R : ^ L'^~^ is an intertwining operator. 

The cocycle b is nontrivial, because in L*^ there are no [/-invariant vectors. 

Explicit expressions for ^ in different models can be obtained from the formulas for the 
vacuum vector in the spaces of complementary series representations by passing to the 
limit as A — > 0. Namely, 

^ = const in model A, 



|7 



2x 1 



-n 



1 H — —j in model B, 

^ — 1^1 ~ Ki-n (V2 in the commutative model; 

here Kp{x) is a Bessel function, see [51, Vol. 2]. In particular, ^ = e"^'^' for n = 2. 

Remark. For A = n— 1, a iC-invariant vector is invariant with respect to the whole group 
0(n,l). 
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Further, we have 

b'{Lo,g) ^ [ \og{l- {uj,uj')) \io' (3 + Sl^-^'dco' in model A, 

Js 

b\uj,g)^ log(|7-7'|) • I - ^^/13 +7 ^23 +^/33r"''c^7 in model B. 

2.8. Restriction to the maximal parabolic subgroup P. 

Proposition 2.7. The restrictions of the complementary series representations of the 
group 0(n, 1), n > 2, and of the special representation ^ j^n-i ^j^j^^ group to the 
maximal parabolic subgroup P are irreducible and pairwise equivalent. In the commutative 
model, an intertwining operator t : L'^ ^ is given by (r(/?)(^) = |^|'^'/^(^)- 

The unitary representation of P arising in this way is a special representation of this 
group. 

Proof. In the commutative model, the representations T^, including the special represen- 
tation T*^, are realized in the Hilbert spaces with the norm 

It follows from the explicit formulas (2.17) for the operators of the subgroup P that r is 
an intertwining operator ^ L^. □ 

Remark. In order to describe the extension of this special representation of P to an arbi- 
trary complementary series representation T^, it suffices to determine only the operator 

'T'A 

2.9. The study of the separabihty of the special representations of the groups 
0(n, 1) and P from the identity representations in the Fell topology. By definition, 
a unitary representation T of a locally compact group G in a Hilbert space H contains 
almost invariant vectors if for every compact set K (Z G and every e > there exists a unit 
vector h C H such that \\Tkh — h\\ < e for all k E K. 

The set of all representations of G that contain almost invariant vectors coincides with 
the intersection of all neighborhoods of the identity representation of G in the Fell topology 
on the set of unitary representations of G. 

Proposition 2.8. 

1) The special representation T of the group 0{n, 1) does not contain almost invariant 
vectors. 

2) The restriction T' of the representation T to the maximal parabolic subgroup P 
contains almost invariant vectors. 

Proof. 1) The representation T of the group 0(n, 1). Let K be the maximal compact 
subgroup of G = 0{n, 1). Since T is a special representation, the space H does not contain 
K-invariant vectors, so that 

/ {Tkh,h)dk^O 
Jk 
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for every unit vector h e H. Integrating the equation 

\\Tkh-hf = 2-{Tkh,h)-{h,Tkh) 

over K, we obtain ||Tfc/i — /ip dk = 2. Hence T does not contain almost invariant vectors. 

2) The representation T' of the subgroup P. Let us use the commutative model of the 
special representation of 0{n, 1). Set 

MO = x.(l^l)-|Cl' (A>o), 

where is the characteristic function of the interval (0,77). Obviously, h{^) G H for any 
Tj > and A > 0. Let us check that for every compact set K C P and every e > 0, and for 
all sufficiently small 77 and A, 

\\Tl^h-h\\^ <€• \\hf for keK. 
Let us write elements k G K ui the form k = ^(7) • diag(r~-^, w, r). Then 

Without loss of generality we may assume that \r\ < 1 on K. Under this assumption, we 
have Xn{H\) Xn{\i\) = X?(l^l) for |^| < 77, whence 

\\Tlh - hf = 2 / {h{i) - {Tlhm) MO \i\ 

This immediately implies the required assertion. □ 

Remark. The proposition remains valid for the special representations T of the group 
U{n,l) considered in §6. 

2.10. Remark on representations in real spaces. In models A and B, the operators of 
the representation T'^ preserve the spaces of real-valued functions. Therefore, the spaces of 
complementary series representations and the space of the special representation, regarded 
as linear spaces over M , split into direct sums of two invariant real Hilbert subspaces: the 
spaces of functions with real and purely imaginary values, respectively. The representations 
of the group 0(?t-, 1) in these real Hilbert spaces are irreducible and equivalent. 

In the commutative model, the representation space splits into a direct sum of invariant 
subspaces over R: the space of functions f{^) + /(— ^^id the space of functions f{^) — 

7R)- 

2.11. Application of the commutative model of the special representation of 
the group 0{n, 1) to constructing an irreducible unitary representation of the 
current group 0(n, 1)^. The role of the commutative model of the special representation 
of 0(n, 1) manifests itself in that it leads to a natural construction of an irreducible unitary 
representation of the current group 0(n, 1)^, i.e., the group of measurable maps X 
0{n, 1), where X is a space with a finite measure m(x), with pointwise multiplication. 

We start from the space L"^ {\^\^~^ d^) , ^ € E""-^, in the commutative model of the 
special representation of 0(n, 1), and construct a cr- finite measure v in the space = 
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$(X) of vector distributions ^{x) = {^i{x), . . . , ^n-i{x)) satisfying the following invariance 
properties: 

1) The measure i/ is invariant under the action of the group 0{n — 1)^, i.e., 

dv{^u) = du{^) for every u{x) G 0{n — l)-^ . 

2) The measure u is projectively invariant under the multiplication by any bounded 
Borel function e{x) G (M*)^ for which the integral J-^ \€{x)\ dm{x) converges, 
namely. 

In particular, du{e^) = dv{^) if log |e(x)| dm{x) = 0. 

In view of these properties, we call v the Lebesgue measure on 

We construct a unitary representation of the group in the Hilbert space L'^{i^) of 
functions /(^) on By analogy with the commutative model of the special representation 
of 0(n, 1), elements 7(x) G act as multiplicators: 

and the action of elements d = (e~-^, w, e) € D-^ is given by the formula 

It is easy to check that these operators generate a representation of the whole group P^. 
The unitarity of Tj is obvious, and the unitarity of follows from the Lebesgue properties 
of the measure u. 

The unitary representation of the group P-^ constructed in this way is irreducible. As 
proved in [18, 19], it can be extended to a unitary representation of the whole group 
0{n,l)^. 

Part 2. Representations of subgroups of the group U{n, 1) 

§ 3. Representations of the Heisenberg group H and their extensions to the 

groups Po = H\Do and P = H\D 

This and the next sections are devoted to the description of the complementary series 
representations and special representations of the group C/(n, 1). We begin with a con- 
struction of representations of the maximal unipotent subgroup, i.e., the Heisenberg group 
H C U{n, 1), and its normalizer, the parabolic subgroup P. 

3.1. Description of the irreducible unitary representations of the Heisenberg 
group. It is known (see, e.g., [50]) that the irreducible unitary representations of the 
Heisenberg group H break into two classes: one-dimensional representations, depending 

on 2(n— 1) real parameters, and infinite-dimensional ones, depending on one real param- 
eter p ^ (the Planck constant). Let us describe the Bargmann model of the infinite- 
dimensional irreducible representations. 

The irreducible representation with parameter p > is realized in the Hilbert space 
H{p) of entire analytic functions f{z) in z = {zi, . . . , Zn-i) G C"""^ with the norm 

f 1/(^)1' e-HN'dM(z), (3.1) 
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where \z\^ = zz* = + ... + |^n-iP and dfi{z) is the Lebesgue measure on ^ 
normahzed by the condition 
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The operator corresponding to an element h = (to, -^o) € H has the form 

Thf{z) ^ eP^'*°-^^''^'-''*o^ f{z + zo). (3.2) 

It is easy to check that these operators are unitary and satisfy the group property. 

The irreducible representation with parameter p < is realized in the Hilbert space 
7i(yo) of entire antianalytic functions f{z) in z — {zi, . . . , Zn-i) € C"^"^ with the same 
norm (3.1). 

The operator corresponding to an element h = {to, zq) G H has the form 

Note that the transition from the space H{p) to the space 7i{—p) reduces to replacing 
f{z) by 

Proposition 3.1. The representations of the Heisenberg group H in the spaces 'H{p), 
p ^ 0, are irreducible and pairwise nonequivalent; they exhaust all infinite- dimensional 
irreducible unitary representations of H. 

Proposition 3.2. The monomials = z^^ . . . z^'Ti and z^ = z\^ . ■ ■z'n-i ^ ^ ^ ^+~^ 
form orthogonal bases in the spaces 'H{p) for p > and p < 0, respectively; their norms 
equal 

where 

\k\ ^ ki + . . . + kn-i, k\ = ki\ . . .kn-i\. 
In this basis, the operators Th, h = {to, zq), for p > have the form 

T^^k^^p{ito-\\zo?) J2 akm{p,zo)z^, (3.4) 

where 

aM = >^ p^_^:^^^, - (3.5) 

For p < 0; the formula for is analogous. 

Remark. The change of variables z ^ z' — \p\^/'^z determines isomorphisms of Hilbert 
spaces 7i(yo) "^(1) and 7i(p) H{—1) fox p > and /> < 0, respectively. Thus every 
irreducible infinite-dimensional representation of H with parameter p can be realized in 
the Hilbert space of entire analytic (antianalytic) functions with the norm 



\f{z)\'e-\'rdp{z) 

'c 



2 
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In this realization, the operators of this representation have the form 

Thf{z) = e^(**°-5 l^oP) V/'^^o* f(z + p^/^zo) for p > 0; 
Thf{z) = e'P^^- 5 \p\\zo?-\p\''^zoz* ^ |/?|i/2zo) for p < 0. 
3.2. Another expression for the norm in the spaces 

Proposition 3.3. For every entire analytic function f{z) on C for every p > 0, 

j \f{z)\^e-p\^\'dti{z)=p^-^ j f{z)7{^)eP^^'^*-\^\'-\^'\'Ufi{z)dfi{z'). (3.6) 



n—1 



Analogously, for every entire antianalytic function f{z) on C' 

J |/(^)|2e-l^ll^l'd/i(^) = |pr-i y f{z)J{^e^P^^<''^*-^'^^-^''^'Up,{z)dp,{z'). (3.7) 

Proof. Let us prove (3.6). It suffices to prove this equation for all monomials f{z) = z^. Let 
Ii and I2 be the left- and right-hand sides of (3.6), respectively, for f{z) = z^. Obviously, 
Ii = k\ p~\^\~^+^. On the other hand, let us substitute the power series expansion e^^°^* = 

^ {z'Yz^ of eP^'^* into the expression for I2. Observing that the terms of the resulting 
sum vanish for / 7^ fc, we have 



|fc|+n-l 



/ zH^e-P\'\ dii(z) =^^— If = /c!p-l^l-^+i 



k\ 

Thus Ii — I2. The proof of (3.7) is similar. □ 

Corollary. The norms (3.1) in the spaces H{p) for p > and p < coincide with the 
norms 

|2 



J f{z) f{z') eP(-'-*-l-P-N'P) dp^{z) dfi{z') (3.8) 



and 



respectively. 



= IpI^""^ j f{z) f{z') e i<^r-\A^-W?) dp,{z) dfi{z'), (3.9) 



Remark. On the space of all functions f{z), the norm (3.1) is strictly positive, while the 
norms (3.8) and (3.9) are degenerate. 

3.3. Extension of a representation of the Heisenberg group H to the group 

Po = H\ Do. 

Proposition 3.4. The infinite- dimensional representations of the Heisenberg group H can 
he extended to the group Pq — H\Dq. Namely, in the Bargmann model 7i{p), the operators 
Td corresponding to elements d = diag(e, u, e) € Dq are given by the formula 

Tdf{z) = f{e-^zu). (3.10) 
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Corollary. The restriction of the representation of the group Pq in the space H{p) to 
the subgroup U{n—1) C Dq splits into a direct sum of pairwise nonequivalent subspaces: 
subspaces of homogeneous polynomials in z G C"~^ for p> and subspaces of homogeneous 
polynomials in z for p < 0. 

In particular, the representation space contains a U{n—1) -invariant vector, namely, 
f{z) — const; this vector is unique up to a factor. 

The explicit expression (3.1) for the norm and the expressions for the operators of the 
group Pq in 7i(p) imply the following proposition. 

Proposition 3.5. For every r > 0, the map f{z) f{r~^ z) is an isomorphism of Hilbert 
spaces TL{p) 7i{r~'^ p), which send the operators T^, b € Pq, on 7i(/?) to the operators 
'^dibd~^ on7i{r~^p), where di — disig{r~^ ,e,r). 

3.4. Direct integrals of representations of the group Pq and their extensions to 
representations of the parabolic group P — Po \ Di — H\D. Let us associate with 
each real number A ^ the following direct integrals of the Hilbert spaces H{p): 

/O r+oo 
n{p) \p\^-Up; H{p)p^-^dp. 
-oo ^0 

In more detail, is the Hilbert space of functions /(/?, 2;) on x C"""^ that are entire 
analytic in z with the norm 

\f{p,z)\^ e-\p\\^\' dp{z)) p^+^-^ dp. (3.11) 

The space is defined in a similar way. 

The unitary representations of the group Pq in H{p) induce unitary representations of 
this group in 7i^. 

Proposition 3.6. The representations of the group Pq in the spaces 7i± can be extended to 
unitary representations of the group P = Pq \ Di. Namely, the operators T^ corresponding 
to elements d € Di are given by the formula 

Tdf{p,z)^f{r'^p,r-h)r^ for d = diag(r-\ e, r), r > 0. (3.12) 

Proof. The unitarity of T^ follows from the definition of the norm in Ti^. Further, Propo- 
sition 3.5 implies that TdTi^T^-i = T^^^-i for any d ^ Di and b e Pq. Hence the operators 
Td, d E Di, and T5, b G Pq, generate a representation of the group P. □ 

Proposition 3.7. The representations of the group P in the spaces 7i\. (respectively, li.^) 
are irreducible and pairwise equivalent. The representations in the spaces n\ and nt. are 
not equivalent. 

Proof. The irreducibility immediately follows from the definition of the operators of the 
subgroup D\ and the fact that the irreducible representations of Pq in the spaces 7i(/?) are 
pairwise nonequivalent. An intertwining operator r : has the form 

r: /(p,^) = H^('^-^)/(p,^). 

□ 
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3.5. The action of the operators Tg, gf € P, on elements of the form fk{p)z^ and 
fk{p)z^. Denote by {LD^ c and {L^)^ c H^, k e the subspaces of functions 

of the form fk{p) and fk{p)z^: respectively. It follows from the definition that 

Let us describe the action of the operators of the group P on elements of — (L^)^. For 
definiteness, we restrict ourselves to the case of the subspaces TC^. 

Proposition 3.8. The action of the operators T+ corresponding to elements g = {to,zo) € 
H and d = diag(e~-'^, w, e) £ D on elements fk{p)z^ G is given by 

^Lzo)^fk{p)z^) = J2'^km{to,zo;p)z^, (3.13) 

m 

where 

( _ ^\\m-s\ k-s -m-s 

^ sl{k — s)\(m — s)\ 

T+{fk{p)z')^k\\e\'e-\''\M\e\'p) ^ Lki{u) zK (3.14) 

\iHk\ 

Here 

rriij 
U ■ ■ 

^«(«) = E (3-15) 

M{k,l) ■' 

and the sum in (3.15) ranges over the set M{k,l) of integer (n—l) -matrices m — 
rriij ^ 0, satisfying the conditions 

^ ^ TTi^j = li , ^ ^ rnij = kj . 

3 i 

The operators T~ act on elements from in a similar way. 

Proof. It follows from the description of the operators Tg in the space H+ that 



= e 

/ „\\m-s\ ^k-s -m-s 



s!(fc — s)!(m — s)! 

This implies (3.13). Further, we have 

T+{fk{p)z>^) = \e\^ fk{\e\'p){e-'zu)f' 

Obviously, 

(rt?.)^ = fc!^l^l J2 Lki{u)z\ 

i\=\k\ 
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where Lki{u) is given by (3.15). This imphes (3.14). □ 

Remark. In § 5 we will show that the representations of the group P in the spaces can 
be extended to unitary representations of the whole group U{n,l). These representations 
of C/(n, 1), called special representations, have nontrivial 1-cocycles. 

3.6. Embeddings of the representations of the groups Pq and P in H{p) and 
into their tensor products. 

Theorem 3.1. For any pi > 0, ...,/?m > there exists a unique embedding of Hilbert 
spaces 

m 

T : n{pi + . . . + Pm) ^ (g) n{pk) (3.16) 

fc=l 

commuting with the action of the subgroup Pq . Namely, 

Tf{z) = F{zi,...,Zn) = f{p~'^J2piZi), where p = J2pi- (3-17) 

In a similar way we define the unique embedding (3.16) commuting with the action of the 
subgroup Pq for any pi < 0, . . . , pm < 0- 

Proof. It follows from Proposition 3.4 (corollary) that the tensor product (^^Li T~^{Pk) 
contains a unique vector invariant under the subgroup Dq. Hence there exists at most one 
embedding (3.16) commuting with the action of Pq. Let us show that (3.17) is the required 
embedding. 
By definition, 

Let us replace the variables zi, . . . , z^, zi € C"""-*^, in this integral with the new variables 

m m 

z = p"^/^ ^ py^ Zi and Wk^^ akiZi, /c = 1, . . . , m-1, 

i=l i=l 

where the vectors = {aki, ■ ■ ■ ,akm), k = l,...,m— 1, together with the vector a = 
{^p~^l'^ P\^ 1 . . . , p~^^'^Pm) form an orthonormalized basis in M"^. We obtain 

||r/f= / |/(/>-V2^)|2e-N^dM.) / e-^r-i^l-^l^n^^("^^) 



P""' / |/(-^)|'e-^N'dM^)^"^"2 



I n— 1 



It follows that r is an embedding of Hilbert spaces. One can directly check that r commutes 
with the operators of the subgroup Pq- D 
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Remark. It is not difficult to see that the representation of the Heisenberg group H in the 
tensor product T^{Pi) is a multiple of the irreducible representation with parameter 

P = J2 Pi- 
Theorem 3.2. For any Ai > 0, . . . , > there exists an embedding of Hilhert spaces 

m 



commuting with the action of the subgroup P. Namely, 

rfip.z) = P-'T.P^^^). (3.18) 

where p = ^Pi- Similarly, for any Ai < 0, . . . , A^ < there exists an embedding of Hilbert 
spaces 

m 

1^ , m^i 



T:nt^Q^ni\ A = ^A„ 

i=l 

commuting with the action of the subgroup P. 
Proof. By definition, 

X f if \f{EPi:P-'EPiZi)\'e-^P^^^^\']Jdfi{zi))l[p^+''^-^dpi. 
In view of Theorem 3.1, it follows that 

The obtained expression can be transformed into 

Note that the second integral equals Hence ||r/|| = ||/||, i.e., r is an embedding. 

The fact that it commutes with the action of P follows from the explicit formulas for the 
corresponding operators. □ 

§ 4. Representations of the groups Pq and P in the spaces H{p) and 

4.1. Representations of the group Pq in the space H{p). Denote by H{p), p 7^ 0, 
the Hilbert space of all functions f{z) on C"^""^ with the norm (3.1): 

11/11' = |pr' / \f{z)\'e-\p\\^\'dp{z). 



'c 



n-l 



Note that H{p) is spanned by the monomials z^z\ k,l E Z'^ ^ 
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Let us define a unitary representation of the group Pq by the same formulas as in the 
case oin{p), i.e., by (3.2), (3.10) for p > 0, and by (3.3), (3.10) for p<0. 
In particular, ?i{p) is an invariant subspace of H{p). 

Proposition 4.1. Theorem 3.1, concerning embeddings of the subspaces TC{p) into their 
tensor products, translates without any change to the spaces H{p). 

4.2. Decomposition of H(p) into irreducible subspaces. For definiteness, consider 
the case p > 0. Similar results for /? < can be obtained by replacing functions with their 
complex conjugates. 

Denote by m = 0, 1, . . ., the subspace of H{p) consisting of all homogeneous poly- 
nomials in 01, . . . ,Zn-i of homogeneous degree m. 

Obviously, the spaces Km are invariant, irreducible, and pairwise nonequivalent with 
respect to the subgroup U{n—1) C Pq- It is also clear that the spaces Km are mutu- 
ally orthogonal. Note that the lowest weight vector in Km is the vector of weight 
(0,...,-m). 

Definition. Denote by Hmip), m = 0, 1, . . ., the invariant subspace in H{p) that is cycli- 
cally generated by the vectors tfj G Km- 

In particular, Hq(p) coincides with the space 7i(yo) of entire analytic functions in Zi 
introduced above. 

Remark. Let us give another equivalent definition of the spaces Hm{p)- It follows immedi- 
ately from the description of the representation that the operators di = i = 1, . . . , n— 1, 
satisfy the following commutation relations: 

diTh = Thdi for ah heH, (4.1) 

n— 1 

di Te,u = Uij dj) Te,u for ah (e, u) € Dq. (4.2) 

Denote by Lm,{p), m = 0, 1, . . ., the Hilbert subspace of functions i/j € H{p) such that 

Pm{di, . . .,dn-i)i/j = 

for all homogeneous polynomials pm of degree m. Obviously, Lm{p) C Lm+i{p)- In this 
notation, we have 

Hm{p) = Lm+l{p) Q Lm{p): m = 0, 1, . . . . 

Proposition 4.2. The representations of the subgroup Pq in the spaces Hmip) ^'^^ 
ducible and pairwise nonequivalent. 

Proof. Assume that Hmifn) is reducible and hence there exists a decomposition Hm{p) — 
Hi@H2 into invariant subspaces. Since Hm{p) contains a unique lowest weight vector, this 
vector belongs to one of these subspaces, say. Hi. But then Km C i^i, whence Hi = Hm{p)- 
Since the lowest weight vectors in the spaces Hm{p) have different weights, these spaces 
are pairwise nonequivalent. □ 
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Corollary. The subspaces H^ip) are mutually orthogonal^ and the space H{p) is their 
direct sum: 

oo 

H{p) = Hm{p). 
m=0 

Proposition 4.3. The restriction of the representation of the group Pq in the space H^ip) 
to the subgroup H is a multiple of the irreducible representation of H with parameter p, 
the multiplicity being equal to dimi^^. The decomposition of Hjn{p) into H -irreducible 
subspaces can be written in the form 

Hmip) = HmM, 
\k\=m 

where H^k{p) is the H -irreducible subspace generated by the vector € K^. 

4.3. The orthogonal basis in Hm{p) associated with the action of the Heisenberg 
group H. Set 

f,{z)=zP = zP,\..zl-i: peZ^-K (4.3) 
It follows from the definition of the norm in H{p) that 

(4.4) 

Let Lp{p) C H{p) be the iJ-invariant subspace cyclically generated by the vector fp{p). 
By Proposition 4.3, the subspace Lp{p) is irreducible, and 

Hm{p) = Lp{p), H{p) = Lp{p). 

\p\=m pez"~^ 

Let us define an orthogonal basis in each subspace Lp{p). By definition, the action of 
the operators T/j, h — {t,z) E H, on the vectors fp is given by 

nfp{z) = {z + areP^''-'f-^-*\ 

where zP ^zl'-.-zl'Ti. 

Definition. Let us define functions fp^q{z), p,q E as the coefficients of the following 

expansion: 

(z + a)^e-^-*= Yl fp,ci{z)a\ a^ = af...C-t, (4-5) 
or, which is equivalent, as the coefficients of the expansion 

^b{z+aY-pza* ^ J2 {p\)-^fp^q{z)lfofl. (4.6) 

In particular, fp^Q = fp. It follows from this definition that 

1) fp,q are polynomials in Zi and Zf, 

2) for every fixed p, the functions fp^q linearly span Lp{p). 
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Proposition 4.4. The polynomials fp^q are given by the following explicit expression: 

where the sum ranges over s € Z'^^ such that s ^ p, s ^ q. 

Indeed, we have 

^b{z+a)*-pza* ^ ^bz* ^{h-pz)a* ^ ^""^ (6 - p zYofl 

r ' q 

' r\s\{q-s)\ s\{t) - s)\{q - s)\ ) 

r,a,q ^ ' p,q s ^ ' ^ ' 

This imphes (4.7). 

Theorem 4.1. T/ie polynomials fp^q, \p\ — m, form an orthogonal basis in Hm[p), m = 
0, 1, . . .; and 

\\fp,,f = ^\\fpf for any p,q. (4.8) 

Proof. The orthogonahty of fp^q and fp/^q/ for p ^ p' fohows from the mutual orthogonal- 
ity of the spaces Lp{p). Further, substituting the expression for T^fp into the equation 

{Thfp,Thfp) = ifpjp), we obtain 

J2(fp^^^ u,,') ^ = ^fv^ fp) = (fp^ fp) E ^ 

q,q' 1 
This implies (4.8). □ 

Corollary. For any p,q G Z^"^, 

Wfpqf-^/^'-'^- (4.9) 

4.4. Formulas for the operators of the group Pq in terms of the basis {fp q} on 

H{p). 

Proposition 4.5. The operators Tt^a, {t, a) € H, have the form 

Tt,a fp,q = eP^'-"^^ J2 fp^i'^ (4-10) 



where 



Cqq'ip, a)=y — [-p^ aV-«+^ (4.11) 



s 

n—1 



and the sum ranges over all s ^"L^ such that s ^ q and q' — q + s ^ 0. 
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n-1 



Proof. For every 6 € C"^ , we have 
Since 



and e"'''^^* = ^ t^a^F, we obtain 



S! 

s 



q r,s,q' ' ' 

Proposition 4.5 follows. □ 

To describe the operators Td, d = diag(e, u, e), where |e| = 1, w € l/(n— 1), we use the 
polynomials introduced above in § 3.5: 

E (IIf!)' P'5^^r'> 

where the sum ranges over the set M(p, g) of integer (n— l)-matrices m = H^TT-ijH, "rriij ^ 0, 
satisfying the condition 

''^kj — Pj for all j; '^^hj — Qi for all i (4-12) 

i j 

(Louck polynomials). Note that these polynomials are the coefficients of the expansion 

e^^^b ^Y,Lpq{x)aibP, a,6€C"-\ (4.13) 

P,Q 

where axb = Y17j=i ^i^ij^j- 

Obviously, Lpq{x) — Lqp{x'), where the prime denotes transpose. 

Note that Lpq{x) can be written as a sum over the subgroup Mq of all integer matrices 
/ for which all rows and all columns sum to 0: 

where is an arbitrary fixed integer matrix satisfying (4.12). 

Proposition 4.6. The action of the operators Td, d = diag(e, w, e), where |e| = 1, w € 
?7(n— 1), on the basis vectors is given by the formula 

Te,u fp,q = P !el^-^l E ^' ■ ^fP'(^) ^«^'(^) fp'^'i'- (4.14) 

p',q' 



38 

Proof. First of all, we have 

Y,iTe,efp,q)a'^^T,,e{z + a)Pe-P'''* = {ez + a)Pe-P''-* 
q 

= el^l {z + eaf e -P'^^* = e'^'l ^ fp,q (ea)« = e'^-^l ^ /p,, 

Hence 

Te,efp,,^e\'^-P\fp,,. (4.15) 
Further, for every u € f/(n— 1) we have 

p' p',q' 

= P! q' ^- Lppf{u) fp'^q' Lqfq{u')a'^. 

p',q',q 

Therefore 

Tl,u fp,q = P ! 5^ ! Lpp>{u) Lq>q{u') fp>^q>. (4.16) 

Combining (4.15) and (4.16) and taking into account that Lq/q{u') = Lqq/{u), we obtain 
(4.14). □ 



^pq- 

D'^{x) = \\Dpq{x)\\\p\^q\=rn, whcrc Dpq{x) = {p\q\f/'^Lpq{x), 



Remark. Observe the following property of the polynomials Lpq. The matrices 

D'^ix) 
satisfy the relations 

D'^{xy) = D'^{x) D'^{y). (4.17) 
Relation (4.17) remains valid if we replace {p\q^^^^'^ in the definition of Dpq{x) by 

YYiZi vT^i"^ ^ where ai + f3i = 1, i = 1, . . . ,n— 1, in particular, if we replace (pJg!)"^/'^ by p\ 
or q\. 

4.5. Representations of the group P in the spaces H^. By analogy with the spaces 
we define spaces as the following direct integrals: 

H^= f H{p)\p\^-Up- Hl= f H{p)p^-^dp. 

J —oo J —oo 

In more detail, is the Hilbert space of ALL functions /(p, z) on R+ x C"^""^ with the 
norm 

ll/f = \f{p,z)\'e-\p\\^\'dp{z))p-+'-'dp. (4.18) 

The space is defined in a similar way. 

The unitary representations of the group Pq in the spaces 7i(p) induce unitary repre- 
sentations of this group in the spaces H\. 
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Proposition 4.7. The representations of the group P in the spaces can be extended to 
unitary representations of the group P — Po \ Di. Namely^ the operators corresponding 
to elements d E D are given by the formula (3.12); 

Tdfip, z) = /(r^p, r~'^z) r^ for d = diag(r"^, e, r), r > 0. 

Proposition 4.8. The spaces decompose into a direct sum of pairwise nonequivalent 
P -irreducible subspaces: 

oo 
m=0 

where 

/O rcx) 
H{p)\p\^-' dp, {Hl)m = / H{p)p^-' dp. 
-oo ^0 

Corollary. The representations of the group P in the spaces {H^)m (respectively, {H^)m) 
are pairwise nonequivalent. 

Proposition 4.9. Theorem 3.2, concerning embeddings of the spaces into their tensor 
product, translates without any change to the spaces H^. 

4.6. Another realization of representations of the groups Pq and P. 

Proposition 4.10. Every Hermitian functional in the space of functions f{z) on C""""^ 
that is invariant with respect to the operators (3.2), (3.10) (for p > 0) of the group Pq can 
be written in the form 

$(/) = J f{z)7{^eP^^'^*-\'\"-\^'\"^Q{p\z-z'\)dfi{z)dfi{z'), (4.19) 

where the kernel Q is an arbitrary (generalized) function. 

In particular, if Q is the delta function, then this functional coincides, up to a factor, 
with the squared norm in the space Hj^[p). 

Assume that $ is a positive definite functional and denote by H{Q, p) the Hilbert space 
of functions f{z) on C"~^ with the norm = ^(/). Formulas (3.2), (3.10) define a 

unitary representation of the group Pq in this space. 

In a similar way we define a representation of Pq in -ff(Q, p) for p < 0. 

Proposition 4.11. If H{Q,p) contains all monomials z^z\ k,l e and is generated 

by them, then the representation of the group Pq in H{Q,p) is equivalent to its represen- 
tation in H{p). 

Further, let us introduce the following direct integrals of the Hilbert spaces H{Q,p): 

roo 

HiiQ)^ H{Q,p)p'^+^-Up. 
Jo 

The representations of Pq in the spaces H^{Q) can be extended to unitary representations 
of the group P by the same formulas as for the space H^. 



40 

Proposition 4.12. For all Q satisfying the positivity condition and the assumptions of 
Proposition 4-^! ^^c? for all X, the representation of the group P in the space is 
equivalent to its representation in the space 7i^. 



Part 3. Representations of the group C/(n, 1) 

§ 5. Models of the complementary series of irreducible unitary 
representations of the group U{n, 1) 

Every complementary series representation is determined by a real number A from the 
interval < A < 2n. Let us describe several models of these representations. 

5.1. Model A: realization in a space of functions on the unitary unit sphere 

S = 5^^-^ in C". The unitary unit sphere in C", 

S : \(JJ\ = IcUll^ + . . . + = 1, 

is a homogeneous space of the group [/(n, 1). Namely, if this group is realized in matrix 
model a (see § 1), then its action on S is given by the formula 



ujg = {uj(3 + 5) ^ {uja + 7) for 9 = 



The complementary series representation with parameter A acts in the Hilbert space 
of functions f{uj) on S with the norm 

= /" |1 - (a;, a;') f{uj) f{io') dco dco' , (5.1) 

JsxS 

where {uj,u)') — ujiuji + . . . + oun^^'n and dw is the invariant measure on S. 

Remark. In the interval n < X < 2n the integral should be understood in the sense of the 
regularized value, see [17]. 

The operators Tg,g—^ ' ^^^^ representation are given by the formula 

Tg^f{u;) = f{u;g)\u;P + S\^-'^. (5.2) 

The group property of these operators is a consequence of the following relation for the 
function b{uj, g) = \ujP + S\: 

b{^,gm) ^ b{u;,gi)b{u;gi,g2) for any gi,g2^U{n,l) 

(the 1-cocycle property). The unitarity follows from the relations 

|1 - (a;,a;') I = |1 - {ujg,uj'g) \b{uj,g)b{uj' ,g) 



and 



duj 
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Proposition 5.1. Complementary series representations and of the group U{n, 1) 
are equivalent if and only if X + fi = 2n or \ — (i. 

An intertwining operator R'^ : ^2n-A form 



f \l-{oJ,uj)'\-^f{cu)du;'. 
Js 



{R 

Is 

5.2. Model B: realization in a space of functions on the Heisenberg group H. 

In model B we use matrix model b of the group U{n, 1). 

The complementary series representation with parameter A is realized in the Hilbert 
space L'^ of functions f{h) — f{t, z) on the Heisenberg group H with the norm 



||/||2= / R-^{h,h') f{h) f{h')dhdh', (5.3) 
JhxH 

where R{hi, h^) = \{hih2^)^i\ and dh = dtdji^z) is the invariant measure on H. 
In the coordinate form: 

R{h,h') = IC + C' + '^'^'I = \i{t-t' + lTa{zz'*) - ^\z-z'\\ (5.4) 

where C, — it — 

To describe the operators of this representation, let us define an action of the group 
U (n, 1) on H by the formula hg = h' , where h' ^ H \s determined by the relation hg = h*h' , 
b* € sPs (the subgroup of upper block triangular matrices). In the coordinate form: 

{C,z)g^{C',z'), 

where 

C' = {Cm + zg23 + gss)'^ (Cfi'll + zg2i +g3i), (5.5) 

z' = (Cfi'ls + zg23 + 933)'^ {Cm + zg22 + 932)- (5.6) 

Further, set 

P{h,g) ^ [i{C,z,g) ^\C,gi3+ zg23 + g33\ for any h&H, geU{n,l). (5.7) 
The operators Tg of the group t/(n, 1) in the space L'^ are given by the formula 

Tg^f{h)=f{hg)(3>^-'-{h,g). (5.8) 

In particular, 

r4.o/(C, z) = /(C + Co - zz^z + zo), (Co, ^o) e H, 
T^,uf{C. z) = /(|e|-'C, -e-^zu) (e, u) E D, 

Ts'f{C,z) = f{i/C,z/C)\C\'-'''. 

The group property and the unitarity follow from the relations 

R{h,h')^R{hg,h'g)P{h,g)f3{h',g) 

and 

d(h.n\ „ 

= r'"(/^,y). 



dh 
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Proposition 5.2. An intertwining operator : — > L^" ^ for equivalent representa- 
tions has the form 

A^f{h)= [ R-^{h,h')f{h')dh. (5.9) 

JH 

Thus the following relations hold: 

r^2n-X j^X _ r^X ^^^^y g ^ ^^^^ 

||/|U = c(A)||AV||2n-A, (5.11) 

that is, 

R-^{hi,h2) ^c{\) [ R-^{hi,h[)R-^{h2,h'2)R^~'^''{h[,h'2)dh[dh'2. (5.12) 

JzxZ 

(The integral is understood in the sense of the regularized value.) 

5.3. Relation between models A and B. In matrix model 6, the stationary subgroup 
of the point ujq — (1, 0, . . . , 0) € S coincides with the subgroup P* = sPs of upper block 
triangular matrices, i.e., S = P*\U{n, 1). It follows that the group H is a section of the 
fiber bundle r : U{n, 1) — > 5 over the punctured sphere, 

r{h) — uJoh, h E H. 

Set 

ip{h) = f{Th). 

Proposition 5.3. The norm in the space of functions on the sphere S, written in the 
coordinates h — (C)^)? has, up to a constant factor, the following form: 



|2 



[ IC + C' + zz'*\-^ |1 - Cl^"^"* |1 - C'l^"^"" ^{h)^{h') dh dh'. (5.13) 
Jhxh 



Proof. The relation uj = ujqH implies that for h = {(, z), 

1 + C , V2z 

UJl = -, uo — 



1-C' 1-C' 

where uJ = {uj2, . . . , cj^). Therefore 

|l-(u;,u;'>| = 2|C + C' + ^^'*||l-C||l-C'|- 

Further, it is easy to check that the measure duj on S and the measure dh on H are related 
by the formula 

dcj = ll-Cl'^'^dh. 

Substituting these expressions into (5.1), we obtain (5.13). □ 

Comparing (5.13) with the expression (5.3) for the norm, we deduce the following corol- 
lary. 

Corollary. The function F{lo), uj E S, in model A and the function f{h), h E R, in model 
B are related by the formula 

f{h) = \i-(:\^-'^F{Th). 
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5.4. Model C: realization in a space of functions ijj{p-, z) on E X C"-^ Model C of 
the complementary series representations of the group C/(n, 1) is obtained from the previous 
one by passing from functions f{h) = f{t, z) in model B to functions 2;) on E x C"^~^, 
where 

/+00 
e-'P'f{t,z)dt. 
-00 

According to the inversion formula for the Fourier transform, we have 

r+00 I I 

fit, z) = i^r-i / ^{p, z) \pr^ e'p'--f 1^1' dp. 



To describe the space L'^ and the operators of the representation in this model, set 

.1*1 



Qx{t)=t ^ e^Ki^x (— ), where i^iy is a Bessel function, (5-14) 



{Z1Z2 for p> 0, 
(5.15) 
Z2zl for p < 0, 

Rx{p, zi,Z2) = elpl(«p(-i.-2)-l-il'-N2p) g^d^i _ ^2|2). (5.16) 

Proposition 5.4. In model C, the complementary series representation of the group 
U{n, 1) is realized in the Hilbert space of functions '^{p,z), p G z € , with the 
norm 

2^1/2 r+00 



where 



2 = |p|2--2 f Rx{p,Zi,Z2)^{p,z)^^^d^l{zi)d^l{z2). (5.18) 



The operators of this representation are given by the formulas 



T,^^^,Xp,0)=e^^*°-|^l(^+«-(^O'^))V'(p,^ + ^o) for ito,zo)eH, (5.19) 
X , , ('^{p^^zu) for d = diag(e,M,€) e Do , 

[^r'^'j/^(r /?, r z) for d = diag(r ,e, r)€Di, 



/i-oo r-\-oo \p\\z\ 
-oo J—oo >> 

(5.21) 

where ( = it — . 

Proof. The formulas for the operators of T"^ immediately follow from the corresponding 
formulas in the previous model. 

Let us derive formula (5.17) for the norm in model C. 

It follows from the definition of the norm in model B that 

= J F{pi,zi;p2,Z2)i/j{pi,zi)ip{p2,Z2)dp{zi)dp{z2) \piP2\'^~^ dpidp2, 
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where 



"+00 r+cx) 

X 



/-too r-\-oo 1 
/ e^(Pi*i-^2*2) - t2 + Im(^i^|))2 + - 1^1 - ^2^"^/' dh dt2 
-00 J— 00 ^ 



'0 

According to [52, Vol. 1, p. 11, formula (7)], 

r+°° 1 7rV2 1^1 

/ (t^ + a^)— 2 cosptdt^--—j- CfrKMp\)- 
Jo r(z/+2) 2a 

Therefore 

F{pi,zi;p2,Z2) 

i (A/zj |2l — Z2\ 2 2 

= S{pi - P2) e^^^'^^'^^) Qx{\p\ \zi - Z2\^) \p\^-\ 



where 



s{p,zi,Z2) = -ipilm{ziZ2) - + \Z2\'^) - ^ - Z2\'^. 



To prove (5.17), it suffices to observe that 

( p{z2zl - \zi\'^ - \Z2\'') forp>0, 

s{p,zi,Z2) = \ , , ,2 , ,2, , n (5-22) 

- - |z2r) forp<0. 

The proposition is proved. □ 

5.5. The spaces L^{p) and their decomposition into Pg-invariant subspaces. In 

what follows, we will use model C of the complementary series representations. 

Definition. Denote by and the subspaces of functions ip{p, z) € concentrated, 
with respect to p, on the half-lines p > and p < 0, respectively. 

By definition, and are Hilbert spaces with the norms 

r+00 rO 

ml- Mlp'-'dp and \mi^ \mi\p\'-'dp, 

Jo J-00 

and the space is their direct sum: 

It also follows that and are direct integrals of the Hilbert spaces L^{p) of functions 
iIj{z) on C^"-^ with the norms \\ilj\\p: 

r+00 rO 

L\= L^{p)p^-^dp and L^-= L^{p)\p\^-'^ dp. 

Jo J-00 
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The explicit formulas for the operators of T'^ in model C imply the following proposition. 

Proposition 5.5. 

1) The subspaces and L'^ are invariant under the subgroup P. 

2) The subspaces L^{p) are invariant under the subgroup Pq. 

According to (5.20), (5.19), the formulas for the operators T^^, d = diag(e, w, e) € D, and 
Tfi, h = {to, zq) € H, do not depend on A and have the form 



2 



Tdil;{z)^tl;{ezu), Th^{z) 



gP^zto — ^ — J ^(2 + ^0) for/?>0, 
,ipto-\p\0-^-zz*^) ^(^ + for p < 0. 



Let us compare the representations of the group Pq in L/^{p) H{p): where H{p) is 
the space introduced in Section 4.1. 

According to Proposition 4.2, the space H{p) decomposes into a direct sum of pairwise 
nonequivalent Po-irreducible subspaces Hm{p)'- 



00 



m=Q 

Proposition 5.6. The representations of Pq in the spaces L^{p) and H{p) are equivalent. 

Proof. Note that the space L^{p) is linearly generated by the same polynomials in z and 
2 as H{p), and the formulas for the operators of the subgroup Pq coincide. It follows that 
L'^ip) is equivalent to a subspace in H{p). The isomorphism of L^{p) and H{p) follows 
from the fact that the norm on L^{p) is non degenerate. □ 

Corollary 1. The space L^{p) decomposes into a direct sum of pairwise nonequivalent 
Po-irreducible subspaces L^{p): 

00 

L\p) = Liip), 

m=0 

where L^{p) is the preimage of H^{p) under the isomorphism J : L^{p) — > H{p). 

Corollary 2. The intertwining operator J : L^{p) H{p) is a multiple of the identity 
operator on each subspace L^{p), i.e., 

Jf ^ Cmf for every f G Lm{p), 
where Cm does not depend on f . 

In view of the isomorphism L'^{p) H{p), as an orthogonal basis in L'^{p) we can take 

the same vectors fp,q{p, z), p,q G as in H{p) (Theorem 4.1). It was mentioned there 
that 

\\fp,qf = ^ fo^ any p, q, where fp = /p,o- 
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Proposition 5.7. The norm of the vector fp in the space L^{p) equals 

" ""^ r(|p|+„-i)r2(A) ^ 

For a proof of formula (5.23), see Appendix 2. This formula implies the following corol- 
lary. 

Corollary. The coefficient in the definition of the intertwining operator J : L^{p) — > 
H{p) is given by 



r(m + n-|)r2(A) 
^ r(n- A)r(m + ^) 



2 



d = c ' ':.J'\ (5-24) 

where c does not depend on m. 

Proof of the corollary. By definition, = jjjjy^ for every / € L^{p), where ||/||^ and 

II /II I are the norms of / in the subspaces Hm{p) and L^{p), respectively. Let f = fp, 
\p\ = m, i.e., 

(zP for p > 0, 

fp — i 

[zP for p <0. 

In this case, (5.24) immediately follows from the expression (5.23) for || fp ||^ and the 
equality \\ fp\\^ ^ p\ p-\p\ . □ 

5.6. Restriction of the complementary series representations to the subgroup 

P. The isomorphisms L^{p) H{p) induce an isomorphism of Hilbert spaces 

/+00 r+oo 
L^{p)\p\^-^ dp — > H^= H{p)\p\^-^dp 
-oo J —oo 

commuting with the representation of the subgroup Pq. It follows from the explicit formulas 
for the operators of the subgroup P — Pq \ Di that this isomorphism commutes with the 
representations of the whole group P in these spaces. Recall that decomposes into a 
direct sum of pairwise nonequivalent P-irreducible subspaces: H'^ — ^^^Q{H^)m- This 
implies the following proposition. 

Proposition 5.8. The subspaces L;^ and decompose into direct sums of pairwise 
nonequivalent P-irreducible subspaces 

oo 
m=0 

where 

roo rO 

lA-1 



/•oo nV 

{LX)m = / L^(p) dp, iL^_U = / Liip) \p[ 

Jo J-oo 



dp. 



In view of the isomorphism — > H^, we can transfer the action of the whole group 
U{n, 1) from to H^. 

Definition. The realization of the complementary series representations in the spaces 
will be called the Bargmann model of these representations. 



47 

We should mention two particular features of the Bargmann model: 

1) In the Bargmann model, it is essential that the irreducible Po-invariant subspaces 
H{p) occurring in the decomposition of do not depend on A. 

2) In the Bargmann model, the action of the operators of the subgroup P is defined 
from the beginning: hence in order to describe a representation of the whole group 
U{n, 1), it suffices to define only the action of the operator Tg. 

To describe the operators Tg in the Bargmann model, we must use the decomposition 
— 0^=o((-^+)'« ® (^-)m) of the space L'^ into pairwise nonequivalent P-irreducible 
subspaces. Let G (^±)m be the components of a vector / G in the subspaces (L^)^- 
Let us identify the elements from and H"^ and denote by and the corresponding 
operators T^. Then the action of in H'^ is given by the formula 

m 

where the coefficients Cm are given by (5.24). 

5.7. Vacuum vectors and spherical functions. 

Proposition 5.9. Up to a factor, the vacuum vector f\ in the space has the form 
f\{uj) = const in model A, (5.25) 

f^(t, z) = {t^ + (1 + ^-f)^-"" in model B, (5.26) 

/a(p,^) = H-^ e^l^l'(l + M!)^-ni;C^_^(|p|(l + M!)) in model C. (5.27) 

Proof. For model A, the assertion follows from the explicit formulas for the operators of 
the representation. When we pass to model P, the vector const goes to the vector (5.26). 
Finally, when we pass to model C, the vector (5.26) goes to the vector 

fx{p,z) = e^l-l' e-^P\t^ + (1 + \^f)h-^dt. 

According to [52, p. 11(7)], the obtained expression coincides, up to a factor, with (5.27). 

□ 

Proposition 5.10. In model A of the representation T^, the spherical function ijjx{g) is 
given by the formula 

M9) = \S\-'F{^,^,nA-\S\-') for g ^ (^^ fj , (5.28) 

where F{a,b,c,x) is the Gauss hypergeometric function, see [51]. 
Proof. By definition, 

ipx{9)^cx [ \ioi3 + 5\^-^''\l- {io,io')^-^diodio' for g^ 
JsxS 

Similarly to the case of 0{n, 1), this expression can be reduced to the form 

i/j^(^g) ^ cx / IcJisinhr + cosh rl'^"^" do; = ca (cosh r)'*'"^^ / \1 + acoi\^~^^ dco, 
Js Js 
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where coshr = \6\, a = ^^7. 

Let us write this integral as a power series in a. We have 

|1 + awil^-^" = (1 + aa;i)^-" (1 + aaJi)^"" 

^ ^ r(n-A + fc) - r(n-A + _ 

r(n-|)fc! £j r(^)/! 

Substituting this expression into the integral and taking into account that the integrals of 
the monomials coicJi with k 7^ I vanish, we obtain 

M9) = cx (cosh rf-'" f; , t^''^ {f\u>i\'dw) a' 



Since \u!i\'^ du) — ^^_^^'_^^| (see Appendix 2), it follows that 

Mg) = CA (cosh r)^-2- ^ a^^ = cx (cosh r)^-^- F(n - |, n - fn, a% 

where F(a,b,c,x) is the Gauss hypergeometric function. Since ipx{e) — 1, we have cx — 1. 
Further, in view of the Kummer relations for the Gauss function (see [51]), we have 

F(n - |, n - f , n, a^) = (1 - a^)^"" F(|, f , n, a^). 

This implies (5.28). □ 

Remark. The above relation for the Gauss function is equivalent to the assertion that the 
spherical functions of the representations with parameters A and 2n — A coincide (since 
these representations are equivalent). 

§ 6. Models of the special representations of the group U{n, 1) 

6.1. Definition of the special representations. Similarly to the case of 0(n, 1), the 
special representations of the group U{n,l) arise as the A ^ or A ^ 2n limits of the 
complementary series representations T^. 

Let us describe the cases A = and A = 2n separately. 

For A = 0, the norm ||/||a degenerates on a subspace of codimension 1, and on this 
space we can introduce the norm 

In contrast to the case of 0(n, 1), this norm for n > 1 degenerates on some infinite- 
dimensional subspace Lq <z LP. 

Theorem 6.1. The completion of the quotient space LP/Lq with respect to the norm \\f\\ 
splits into a direct sum of two irreducible nonequivalent subspaces. The representations of 
the group U{n, 1) in these spaces are special representations. 

Note that on the subspace Lq there is a nondegenerate norm, given by 
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and the representation of U{n, 1) in the Hilbert space with this norm is irreducible. 

The space L, which is the A ^ 2n limit of the spaces of complementary series 
representations, has a dual structure. Namely, ||/||a degenerates for A = 2n and n > 1 on 
a subspace L2n of infinite codimension. In the space L2n there is a nondegenerate norm 

which is degenerate on a one- dimensional subspace L^". 

Theorem 6.2. The completion of the quotient space L2n/Lp'^ with respect to this norm 
splits into a direct sum of two irreducible nonequivalent subspaces. The representations of 
the group U{n, 1) in these spaces are special representations equivalent to those defined 
above for A = 0. 

Let us describe several realizations of the special representations. 



6.2. Model A: realization in a space of functions on the unitary sphere 5" C 

as A ^ 0. We assume that the group U{n, 1) is realized in matrix model a. 

Denote by L the subspace of functions /(a;) on the unitary unit sphere 5 C C satisfying 
the condition 



/ f{u;) div = 0, 
Js 



where dco is the invariant measure on S. Let us define an action of the group U{n, 1) on L 
by the formula 

Tgf{u;)=f{u;g)\b{u;,g)\-^^, 

where 



u;g = {uj(3 + 5)-'{uja + ^), b{uj, g) = uj(3 + S for g = (^^ fj 



(6.1) 



The fact that L is invariant under the operators Tg follows from the equality d{ijog) = 
\iop + 5\-^''duj. 

We introduce on L the following scalar products: 

(/i,/2)- = - / \og{l-{u;,u;'))fi{u;)7^dwdu', (6.2) 
JSxS 

(/i,/2)+ = - / log(l-(^JV)")/i(c^)M^c?a;o?a;', (6.3) 
JSxS 



(/i,/2) =(/i,/2) + + (/i,/2)- = -2 / log|l-(a;,a;')l/iH/2(c^0^^^^', (6-4) 

JSxS 

where {to, to') = uJiiD[ + . . . + LVn<^'n- 

Note that (/i,/2) is obtained by passing to the limit from the scalar products in the 
spaces of complementary series representations. 
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Proposition 6.1. The norms ||/||± = {f,f)±'^ can be written in the form 



k>0 ' 



2 



(6.5) 



fc>0 

k , ki , ,k„ 



Here we use the notation k\ — ki \ . . . knl, = A;i + . . . + kn, = cu^^ . . .oj^ 

Indeed, in order to prove, for example, the first equality, it suffices to use the series 
expansion 

log(l-(a,,a,')) = -i:|i|lf'-'^'- 

fc>0 ' ' 

Corollary. 

1) The scalar products (/, /) ± are sign definite on L. 

2) The conditions ||/||+ = and \\f\\- = are equivalent, respectively, to the conditions 

/ Lu'' f{iu) duJ^O and / cu'' f{u) du) ^ for all k > 0. 
Js Js 

Proposition 6.2. The operators Tg preserve the norms \\f\\±, i-c., \\Tgf\\±= ||/||± for all 
geU{n,l). 

Proof. Since + du = d{ujg), it follows that 

IIVII- = - f log(l - {oo,uj'))f{ujg)7(^7^d{ujg)d{uj'g) = 
JSxS 



'SxS 

-1 , ./„-l\ 



/ log(l - {u;g-\ Jg-^)) f{uj) f{co') dw dco'. 
JSxS 



Note that 



1 - {u;g-\u;'g-^) = (1 - (a;, a;')) 6"^^, 9'') ^'H^', Q-')- 

Hence 

II VII- = 11/11- - / {iogb{ijg-')+\ogb{iv'g-^))f{iv)Jii:J^divduj'. 
JSxS 

Since f ^ L, the second term in this sum vanishes. 

For the norm ||/||+, the proof is the same. □ 

Corollary. The following subspaces in L are invariant under the action ofU{n, 1): 

L+ = {feL\\\f\\.^0}; i:- = {/€i| ||/||+ = 0}; Lo = L+nL_. 

Let I/O C 1/ be the subspace of functions / with ||/|| = 0. By definition, the norm ||/|| is 
nondegenerate on the quotient space L/Lq. 

Definition. Denote by jC, jC+, and C- the complex Hilbert spaces obtained by completing 
L/Lq, L_|_/Lo, and L-/Lq with respect to the norm 
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Obviously, 

C = C+®C- 

and 

11/11 = 11/11+ on £+; 11/11 = 11/11- on 
As a result, we obtain the following theorem. 

Theorem 6.3. In model A with X = of the two special representations T+ and T~ of 
the group ?7(n, 1), 

1) The direct sum ofT^ and T~ is realized in the Hilhert space C, the completion with 
respect to the norm ||/|p = —2 J^^^ log |1 — (cj, cj') | /(a;) f {lo') dto dw' of the quotient 
space L/Lq, where L is the space of functions f{uj) on the sphere S = 5^""-^ with 
Jg f{co) duj — and Lq is the subspace of functions with norm 0. 

2) The subspaces C+ and C- of the special representations are determined by the con- 
ditions jC± = {f E C I ||/||=F = 0}, where ||/||+ and \\f\\- are given by Proposi- 
tion 6.1. 

3) The operators of the group U (n, 1) act in £,± as follows: 

T^f{uj)^f{ujg)\h{uj,g)\-^^ (mod Lq), 

where cu^f = (a;/3 + (a;a + 7), b{co, g) — lv/3 + S for g — 

6.3. The structure of the special representations in model A for A = 0. Let us 

describe the subspaces L, L_, and Lq defined in Section 6.2 in terms of generating 
elements. Note that the original space L is linearly generated by the monomials co^co^ = 
Ylcof^cdl\ Let us write k > I \i ki U and k ^ I. Further, set 

Ck= \uj''\'^du}, k e Z^- 
Js 

an explicit expression for will be given below. 

The description of the norms ||/||± implies the following proposition. 

Proposition 6.3. 

1) Ifk> I, then ||cu'=cu^||+ ^ and ||a;^a;^||_ = 0. 

2) Ifl> k, then \\ijo^Lb^\\- ^ and ||a;^a;^||+ = 0. 

3) If k :^ I, I :^ k, k ^l, then ||cfc+;a;'^ - Cfca;^+^a;^||± = ||cfc_|_/a;'^ - Cfca;'a;^+^||± = 0. 
(Here k^leZ^.) 

Corollary. 

1) The subspace L is linearly generated by the monomials uj^u)\ k ^ I, and the binomials 
Ck - cojc^^p. 

2) The subspace Lq is linearly generated by the monomials co^co^, k ^ I, where k ^ I, 
I ^ k, k ^ I, and the binomials Ck+ico^ — Ckoo^^^ob^ Ck+ico^ — Ckoo^ob^^^ k^l eZ'^. 

3) The quotient spaces Lj^/Lq and L-/Lq are linearly generated by the monomials lo^ , 

> 0; and Co^ , > 0, respectively. 

Note that Lq = for n = 1. 
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Proposition 6.4. The monomials co^ and Co^ , k > 0, form orthogonal bases in C+ and 
C-, respectively; their norms equal 

fc„2 _„-fcN2 _ 47r^"A:!(|A:|-l)! 

[(|/c|+n-l)!P 

Proof. The fact that these monomials really generate the corresponding spaces follows from 
Proposition 6.3 (corollary). The orthogonality follows from the explicit formulas (6.5) for 
the norm. The same explicit formulas and (8.6) imply that 



= ii^'ii- = ;;„., 71'' (6-6) 



I, ,ki\2 _ \\.-,ku2 



(I^-l - 1)! 



A;! 

Since 



L 



' (|A;|+n-l)! + 



'5 

this implies (8.6). (For a proof of the latter formula, see Appendix 2.) □ 

Corollary. Elements of the spaces Cj^ and C- can he interpreted as the boundary values 
of regular analytic (respectively, antianalytic) functions in the unit ball in C^. In this 
interpretation, if a function f in the ball is given by the series f{z) — Ylk>o^kZ^ , then 
||f||2_4 2nv- k\i\k\-iy. I |2 

11/11+ -"^TT Z^fc>0 [(|fc|+n-l)!]^ I^'^l • 

6.4. Formulas for the 1-cocycles ?7(n, 1) — > £ and U{n, 1) — > C± in model A. Let 
us find explicit formulas for the nontrivial 1-cocycles on the group U{n, 1) associated with 
the two special representations of U{n, 1) in the spaces jC+ and £.-. 

Note that for A = the nontrivial 1-cocycle a : U{n, !)—>£ = C+ © C- is given by the 
formula 

a{u;,g)=Tgfo-fo = \cop + S\-^^-l for g = (^^ (6.7) 

(since the vector fo{u)) = 1 is invariant under the maximal compact subgroup of U{n, 1) 
and fo^ C). 

Obviously, the 1-cocycles : ?7(n, 1) — > C± associated with the special representations 
are the projections of a to the corresponding subspaces: 

a{uj,g) = a'^iuj^g) +a~{uj,g), a"^ e C±. (6.8) 

Let us find explicit expressions for these cocycles for A = and A = 2n. 

Proposition 6.5. For A = 0, the nontrivial 1-cocycles a^{co,g) are given (modulo Lq) by 
the following formula: 



a^icv, g) = a- (a;, g) = (5^ (a;/5 + (5)"" - 1. (6.9) 
In particular, a'^{co,g) = — for n = 1. 

Proof. Setting 7 = PS~^, we have 

a{uj, g) = (1 + a;7)- (1 + CU7)-" - 1 

k,l 



^-1)! i-i 
— 11 
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Since uj^u} 7^ on only for > Z, it follows that 

fc>0 I \ ^ )• • 

According to Proposition 6.3, on we have 

jk^i-i ^2h±i^k= (fc + Q! (|fe|+n-l)! ^_fc 
Cfc (|fc + /|+n-l)! k\ 

Substituting this expression into (6.10), we obtain 

fc>0 ■ / 

= H-2-((l+a;7)-"- 1) (1-7*7)-"- 
Note that 1 - 7*7 = |(5|-2(|(5|2 - f3*(3) = \5\-'^. Hence 

a+(a;, g) ^ {1 + tv-fy - 1 = 5" (a;/5 + (5)"" - 1. 
The proposition is proved. □ 

Proposition 6.6. For A = 2n; the nontrivial 1-cocycle a'^{uj,g) is given, up to a factor, 
by the following formula: 

a+(a;,^) = e-^"-^-l; (6.11) 
the cocycle a~ is obtained from a"*" by taking the complex conjugate. 

Proof. It follows from the formulas for a+(a;, g) in the case A = and for the intertwining 
operator R:LP^ that 

a+{uj,g)= [ log(l-(a;,a;'))(l+a;'7)-"da;', 
Js 

where 7 = /35~^. Expanding log(l — {uj,lv')) and (1 + ljo'^)~'^ into a power series and 
throwing off the terms with zero integrals, we obtain 

, ^r(-l)l^l(n+|A;|-l)! ^ f ^ ika ^ /l 
The proposition is proved. □ 



6.5. The "strange" irreducible representation of the group [/(n, 1) in the space 

Cq. Besides two special representations, there exists another irreducible unitary represen- 
tation of the group f/(n, 1) that can be obtained from complementary series representations 
in the A ^ or A ^ 2n limit (the so-called "strange representation", see [38, 37, 36]). 
Note that there is no analog of this representation for the groups 0(n, 1). 

Let us construct this representation starting from model A of the complementary series 
representations for A — > 0. 
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We define a scalar product on the subspace Lq = fl L_ by the formula 

'SxS 

Proposition 6.7. The norm \\f\\o = (/i,/2)o'^^ written in the form 



(/i,/2)o=lim^?^%^= / log2|l-(a;,a;')|/i(a;)/2(a;0da;da;'. (6.12) 



Y^CkCil j a;V/(a;)da;|2, (6.13) 
k,l>Q "^^ 



where = ■^^^fep^- swm ranges only over multi-indices k > and I > such that 
neither k > I nor I > k. 

Proof. We have 

log2 |1 - (a;, a;') | = (^ CfcC^'^cJ''^ + ^qcdV^^ 

fc>0 l>0 
k,k'>0 l,l'>0 k,l>0 

Let us substitute this expression into (6.12). Since Lq = nL_, the terms of the first two 
sums make zero contributions. The same holds for the terms of the third sum with k > I 
or l> k. Thus (6.13) holds. □ 

Corollary. The norm \\f\\o is positive definite on Lq. 

Denote by Cq the complex Hilbert space obtained by completing Lq with respect to the 
norm ||/||o. 

Proposition 6.8. The norm ||/||o is invariant under the operators Tg; thus the operators 
Tg form a unitary representation of the group U{n, 1) in the space Cq. 

Proposition 6.9. The representation of the group U{n, 1) in the space Cq is irreducible. 

6.6. Model A of the special representations in a space of functions on the unitary 
sphere for A — > 2n. In model A with A = 2n, the direct sum C = C-^. © C- of the special 
representations is realized in the quotient of L^'^/L2n by the subspace of constants with 
the norm 

= (-1)" / S^^\l-{u;,u;'))f{u;)7W)du;du;', (6.14) 

JSxS 

where 6^'^^ is the derivative of the delta function on the sphere 5", given by the formula 
/^5(1 — {u),u>')) f{u>') du)' — f{uj). The operators of the representation are given by the 
formula 

{Tgf){u;) = fiivg). 

An intertwining operator R from model A with A = to model A with A = 2n has the 
form 

{Rf){uj)^ I \og{\l-{u,J)\)f{J)dw'. 
Js 
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Note that the norm in model A with A = can be written as an integral over the sphere 



S: 

|2 



Js 



's 

In the model under consideration with A = 2n, the subspaces C-^ and £_ coincide with 
the subspaces of boundary values of analytic (respectively, antianalytic) functions in the 
unit bah D C \z\'^ = \zi\'^ + ■ ■ ■ + [z^P < 1- 

The special representations T+ and T~ can be realized directly in the Hilbert spaces of 
analytic and antianalytic functions on D. For this, it suffices to use the following assertions. 

1) The ball D is a. homogeneous space with respect to the action z ^ zg of the group 

U{n, 1), where zg — {u>P + S)~^{Lva + 7). 

2) On D there exists a U{n, l)-invariant measure, namely, (1 — dfi{z), where 
dfi{z) is the Lebesgue measure on D (see [54]). 

Theorem 6.4. The special representation T = T+ (respectively, T = T~ ) of the group 
U{n, 1) is equivalent to the representation in the quotient of the Hilbert space of analytic 
(respectively, antianalytic) functions on the ball D by the subspace of constants with the 
norm 

/ \f{z)\H^^\l-\z\^)dii{z). 
Jd 

The operators of this representation have the form {Tgf){z) = f{zg). 

Note that the vectors A; € Z", form an orthogonal basis in this space, with H^'^P = 
(|fc|+n-l)! 

(|fc|-l)! • 

6.7. Model B: realization in a space of functions on the Heisenberg group H. 

Model B of the two special representations of the group ?7(n, 1) is obtained from model A 

by passing from functions F{uj) on the sphere 5" to functions f{h) = f{(, z), ( = it — 
on the Heisenberg group H according to the formulas 

f{C,z) = \l-C\-'''F{uj), 

where 

^ / 1 + C V^Zi V2Zn-l 

Vi-c'i-C''"' 1-C 

We will describe model B only for A = (for X — 2n the description is similar). In this 
description, we use matrix model b of the group U{n, 1) (see § 1). 

Theorem 6.5. In model B of the two special representations T"*" and T~ ofU{n, 1) with 
A = 0, 

1) The direct sum ofT^ and T~ is realized in the Hilbert space C, the completion with 
respect to the norm 



= - / \ogR{h,h')f{h)f{h')dhdh', (6.15) 

JHxH 

where dh is the invariant measure on H and 

R{h, h') = R{C, z- C', z') = IC + C + ^^*|, (6.16) 
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of the quotient space L/Lq, where L is the space of functions f{h) on the group H 
with f{h) dh — and Lq is the subspace of functions with norm 0. 

2) In the coordinates h = (C, z) on H , the operators of the representation T = T+ ©T~ 
have the following form: 

Tgf{Cz) = f{C',z')\b{Cz;g)\-^^ (mod Lq), (6.17) 

where 

C' = {Cm + zg23 + 933)'^ {Cgn + 2:^21 + gsi), (6.18) 

z' = {Cgi3 + zg23 + 933)'^ {C912 + Z922 + 932), (6.19) 

&(C, z; 9) = C913 + zg23 + 933- (6.20) 

3) The subspaces and £_ of the special representations are determined by the con- 
ditions C± = {/ € £ I ||/||ip = 0}, where \\f\\- and ||/||+ are determined by the 
following formulas: 

" E2'^'-^^^ / B,ih)fih)dh\ 

('6 211 



+ 

fc>0 



where k = (fci, . . . , kn), \k\ = ki -\ h kn, 

B,{h) = B,{C z) = |i±||j^ • z\^ • • • z^i. (6.22) 



Remark. Obviously, the conditions ||/||- = and ||/||+ = are equivalent, respectively, to 
the equations 

/ Bk{h) f{h)dh ^0 and / Bk{h) f{h) dh ^ for k > 0. (6.23) 
Jh Jh 

Proposition 6.10. In model B with A = 0, the nontrivial 1-cocycles : U{n, 1) C± 
associated with the special representations of the group U (n, 1) are the projections to 
and C-, respectively, of the following function a : C/(n, 1) — > C: 

a{C,z;g) ^Tgfo- fo, where /o = |l-Cr^''. 

Proposition 6.4 implies the following result. 

Proposition 6.11. The following functions fkiCi z), k — (fco, fci, . . . , kn-i ) GZl, \k\ > 0, 
form an orthogonal basis in C+: 

fkiC z)^{l-C)--{l- C)-"-l^l (1 + C)'° z'l^ . . . (6.24) 
A similar basis in C- is obtained by replacing ( and Zi with their complex conjugates. 
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6.8. Model C: realization in a space of functions tp{p,z) on E x ^. Model C is 

obtained by passing from functions f{h) = f{(, z), ( = it — on H in model B (with 
A = 0) to functions ip{p, z) according to the formula 

/+00 
e'P^f{t, z) dt. 
-oo 

Proposition 6.12. In model C, the subspace L of functions %l){p,z) is determined by the 
relation 



The norm ||^|| on L is given by the formula 

-+00 



ip{0,z)dp{z) = 0. 



oo 



where 



(6.25) 



(6.26) 



g|p|(ap(.i,.2)-Nip-|.2p) ^(^^ ^(^^ ^2) dii{zi) dfl{z2), (6.27) 



0^(^1,2^2) 



In another way, 



fcez 



n-l 
+ 



|p|2n+|fe|-2 



2:221 /or /?< 0, 
2:1 forp>Q. 



^^fcg-|p|z^*^(^^ Z)d/^(z) 



p2n+|fe|-2 



/c 



— t 

1 2^ e 



^{p,z)dp{z) 



for p<0, 
for p > 0. 



(6.28) 



(6.29) 



Proof. Relation (6.25) is equivalent to relation (6.17) in the realization on H. To calculate 



IV^II, we will use the equation ||?/)| 



lim 



dX 



■, where WtpWx is the norm in the space 



of the complementary series representation. Substituting into this equation the explicit 
expression for HV'IIa taking into account that Qx{t) = const for A = 0, we obtain (6.26) 
and (6.27). Substituting into (6.27) the power series expansion of the function e"^^^^'^^) for 
p> and /? < 0, we obtain (6.29). □ 



Corollary. In model C , the space C decomposes into a direct integral of subspaces: 

"+00 



/-I- 00 
C{p) \p\-Up, 
-00 



(6.30) 



where C{p) is the quotient of the Hilbert space of functions onC^ ^ with the norm (6.27) 
by the subspace of functions with norm 0. 

Theorem 6.6. In model C, the subspaces C+ andC- of the special representations coincide 
with the subspaces of functions ip{p, z) concentrated, with respect to p, on the half-lines 
p > and p <0, respectively. 
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Proof. It suffices to prove that z) € C+ if z) = for p < 0, and, similarly, 
(p{p, z) € C- if (^(p, z) = for p > 0. 

Let us use condition (6.23), which describes when a function f{h) = f{t, z) in model B 
belongs to the subspace £+. Passing from functions f{t, z) to functions '0(/?, z), we obtain 
a condition for if) € Cj^ in the form 



2)2fc'e-|pl^^(-p,2) |pp-idp(i/i(2) = for |/e|>0. 



(6.31) 

where 

A,(p, ^) = y ^ 1^4^ ^ = - (^-32) 

Let us check that this condition is satisfied if '0(/?, z) — Q iox p < 0. Indeed, the function 
A^(p, z) can be written as a linear combination of the integrals 

y_oo (1-C)"^ 7-00 {a-it)rn' ' ' ^ ^ ' 

1^12 

where a = 1 + -i^. It is known that for p > these integrals vanish for every m > 0. Hence 
if V'(/0, 2;) = on the half-line /O < 0, then Ik{ip) = for > 0. For \k'\ = 0, the expansion 
of Ak(p, z) contains the term e~^P^ dt — 5{p). In this case, the corresponding term in 
/fc('0) vanishes because of (6.25). 

In a similar way one can prove that if V'(p, -z) = for p > 0, then -0 G □ 

Corollary. In model C , the spaces and C- decompose into direct integrals of sub spaces: 

r+00 rO 

C+ = / C{p) p-' dp, C- = / C{p) \p\-^ dp; 

Jo J-oo 

the norms in the spaces and respectively, are given by 

r+00 rO 

ml- \mlp-'dp, \m'-^ \mi\p\-Up, (6.33) 
^0 ^-00 

where HV'llp 'is given by (6.27) or (6.29). In more detail, 

E ^ r / ^ "'"'^^^ ^(^' ^) " ipi^^+^i^i-^ dp, 

.t;f_i k\ J-oo Jc"-i 



2 _ 

fcez' 



h r I z'e-f^^'^{p,z)dp{z) 



2 

+ ^ k\ 



\'2n+m-^dp. 



(6.34) 



Formulas for the operators and T~ of the representations of U{n, 1) in and 
respectively, are obtained by passing to the limit as A ^ from formulas (5.19), (5.20), 
and (5.21) for the corresponding operators in complementary series representations. Thus 
we obtain the following theorem. 
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Theorem 6.7. In model C, the operators of the representations ofU{n, 1) in C± have the 
following form. In the space C+ : 

Ttl^Jip.z) = e^(^*o-^-^^o*) V;(p,z + zo), (to.zo) e (6.35) 

T+^i^{p, z) = V'del V, e-hu) , (e, u) G D, (6.36) 

oo oo 

Tt^{p,z)^p--+' I /e-^^+1^|C|-'>'^-V(p, ^)rftV, C = ^^-^• (6-37) 



-oo 



In the space C- 



T,l,^i^{p,z)=eP^'-+'-^+^-^*H{p,z + zo). {to.zo)GH, (6.38) 

T^u'^ip, z) = V^del V, e~^zu) , (e, u) € D, (6.39) 

oo ^ 

T-^{p,z)^\pr+' J |e-^^+t|C|-2-|/rV(/, ^)dtV, C = ^i-^- (6.40) 

— oo — oo 

6.9. Relation to the Bargmann models of representations of the Heisenberg 
group H. Above, by the Bargmann model we meant the reahzation of the irreducible 
representation of the group H with parameter p in the Hilbert space 7i{p) of entire analytic 
(for p > 0) or antianalytic (for p < 0) functions on C^~^ with the norm 

\\ff = \Pr' I \f{z)\^e-\P\\^\'dp{z). 
According to Proposition 3.3, for p > this norm coincides with the norm 

and for /? < 0, it coincides with the same norm in which p is replaced by \p\ and z' z* in 
the exponent is replaced by z{z'Y . 

Also recall that the representation of H in the space ^{p) can be extended to a unitary 
representation of the group Pq. 

Comparing the description of C{p) and 7i(p) and the formulas for the operators of the 
group Pq in these spaces, we conclude the following. 

Proposition 6.13. There is a natural isomorphism of Hilbert spaces 

T : C{p) ^ n{p) 

commuting with the action of the group Pq. Namely, ti/j e ^-ip) is determined for p > 
by the equations 

[ z^e-P^^* il^{z)dp{z)^ j z^e-P^^* Til^{z)dp{z) for every G Z^^'^ (6.41) 
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In other words. 



„|fc|+n— 1 , 

T'il;{z)= V afc^^ where = / e-P''* il;{z) dfi{z). (6.42) 



k\ 



In a similar way we define r for p <0. 



6.10. The Bargmann model of the special representations. Consider the following 
direct integrals of the Hilbert spaces H{p): 

r+oo rO 

n+ = / n{p) p-' dp, H- = / n{p) \p\-' dp. 

Jo J-oo 



In other words, 7i+ is the space of functions f{p,z) — ^j^^^n-i fk{p)z , where fk{p) are 
functions on the half-line p > 0, with the norm 

ll/f = E f IA(P)I'H"'""'^P- (6-43) 

The space 7i_ is defined in a similar way. 

The isomorphisms r : C{p) Ti-ip) induce isomorphisms of Hilbert spaces 

r : £+ ^ n+, C- H- 

commuting with the action of the group P in these spaces. Namely, for if) G C+ we have 

|fc|+n-l r 

T^{p,z)^ J2 fk{p)z\ where fkip)^^^^;^ z^ e-^'^* i,{p, z) dp{z). 

(6.44) 

In a similar way we define r on C-. 

In view of this isomorphism, the spaces 7i± are new models of the special representations 
of the group U{n, 1). Let us call them the Bargmann models of the special representations. 

In the Bargmann model, the action of the operators of the subgroup P is given by 
the same formulas as in model C. The action of the whole group C/(n, 1) is determined by 
indicating only one operator Tg. Let us study the structure of this operator. For definiteness, 
we restrict ourselves with the case of 7i+. 

By definition, the operator on is given by 

Ts'ip{p,z) = T^i{p,z), 

where ii^i{p, z) € stands for the right-hand side of (6.37). 

Denote by 1/^ C 7i+, k G Z^T^, the subspace of functions of the form /(/?) z^. Obviously, 

Proposition 6.14. The subspaces C H± are invariant under the operator Tg. 
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Proof. Let ip € L^, i.e., ilj{p,z) = f{p)z^. Then, according to the formula (6.37) for the 
operator T+ in the space C+, we have T^i(j{pj z) = Tilji{p, z), where 

oo oo ^ 2 

MP: z) = p-'^^' {J I e-^^+T|C|-'"rl'lp'"-V(p')t^*V)^', C = ^*-^. (6.45) 

-oo 

Obviously, the function -01 can be written in the form tpi{p,z) = {Ylo^ fn{p) \z\'^'^)z^. It 
follows from the definition of r that r{fn{p) z^) € for all n. Hence ifji G I/fc. □ 

Denote = for p > 0; = for p < 0. 

Proposition 6.15. T/ie action of the operator on functions of the form p'^z^, a € C, 
regarded as distributions on 7i+, is given by 

Tt(pl z") = (6.46) 

For a proof of (6.46), see Appendix 2. 

Proposition 6.16. The action of the operator Tg on functions ^fc(a; p, z) = pl^l e~"'P z^ € 
L^, Re(a) > 0, is given by 

TsUa; p,z)=a-\^\Ua-^- p,z). (6.47) 
Proof. Let us use the equation 



He-^Pp ^+--1 dp = r(f + zd) a- 2 
Jo 



-la 



'0 

By the inversion formula for the Mellin transform, 

/+00 Ifcl 1/^1 

a-^-^^r(f +m)p^-^^d(7. 
-oo 

According to (6.46), we have 



r(f + ia) 



Therefore 

|fc| 



-oo " ^ 



= a l^lefc(a ^p,^). 

The proposition is proved. □ 

It is convenient to write the action of the operator Tg in passing from functions 
fk{p) to functions Ffc(cr) z^, where 

Fk{cT)^ / /fc(p)p-V---idp. (6.48) 

^0 
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By the inversion formula for the Melhn transform, 

fk{p)=p- / Fk{a)p'''da. (6.49) 

^— oo 

Let us introduce the Hilbert space of functions F{a, z) entire in z € C""""*^ with the 
norm 

/oo 
\Fk{a)\^da for F{a,z)^ ^k{<y) (6.50) 

Proposition 6.17. The map 

where fk{p) and Fk{\) are related by (6.48) and (6.49), is an isomorphism of Hilbert spaces 

The assertion follows immediately from the Plancherel formula for the Mellin transform. 
Denote by T+ the image of the operator on Tt-^. under the map TC-\- 7i+. 

Proposition 6.18. The action of the operator T+ on functions F]^{\) z^ G 7i+ is given 
by 

Tt{F,{X) z^) = (-1)1^1 i.'^^f/ F,(-A) zK (6.51) 

2 



^(^A+f) 



Proof. According to Proposition 6.15, we have 

T+{p 2 +^'^ z^) = i-lp — ^ ^ p 2 zK 

Hence (6.49) implies 

v-oo r(-i<T + i|i) 

On the other hand, 

r/(Ffc(a)^^)pV+-dc7. 

-oo 

Comparing these equations, we obtain (6.51). □ 
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6.11. Description of the nontrivial 1-cocycles : U{n,l) — > H±. The nontrivial 
1-cocycle a : U{n, 1) — > 7i, where H = H+ © is given by the formula 

a{g) = Tgi - C, 

where ^ = f{p,z) is invariant under the action of the maximal compact subgroup U C 
U{n, 1) and ^ ^ L. The 1-cocycles a^{g) are the projections of a{g) to the subspaces H±. 
By definition, a{u) = for u eU, whence 

a{ug) = a{g) for u E U. 

Since an element g G U{n, 1) can be written as the product g — ub, where u G U , b E Pi, 
the 1-cocycle a{g) is uniquely determined by its restriction to the subgroup Pi. 

Theorem 6.8. In the Bargmann model, the function ^ in the formula for the 1-cocycle 
has the form 

C(p,0) = e-l^l. (6.52) 
Proof. We use the fact that in model B the function ^ has the form 

(^(C,0) = |l-Cr'", where C = ^^-^. 
It follows that in model C it has the form 

u|2 r+oo 

^{p^z) = l^r^+i el^l 2 / |l - Cl-^"" e-'P* dt. 

J—oo 

Note that ifj is an even function in p; thus below we assume that p > 0. The required vector 
^ in the Bargmann model for p > has the form 



where the functions fk{p) are given by (6.44). In view of this equation, it is clear that 
fk{p) = for A; > 0. Hence 

/+00 n 
/ eP'^\l-C\-^^dp{z)dt. (6.53) 
-oo JC""^ 

Let us calculate the integral (6.53). We have 
Obviously, 

/ = — dt — ;— —- dt — Res ;— 

7-00 (1-0" J-oo (1 + kP + O" c=-i-i^PMi + kP + 0">' 

= const 6-^(1+1^1'). 

Hence 

( 1 - 4-Yfo{p) = const p'^-i e"^ / e^'^l' d/^(^) = const e"^. 
\ dxJ Jc"~^ 
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Since /o(oo) = 0, it follows that /o(p) = const e p. □ 

Appendices 

§ 7. Appendix 1. Canonical states and canonical representations 

Canonical states on groups and the corresponding canonical representations were first 
introduced for the group 5'L(2,]R) in [10], and then for the groups 0(n, 1) and [/(n, 1) 
in [17]; they were used for constructing irreducible unitary representations of the current 
groups 0(n, 1)^ and C/(n, 1)^. These representations are unitary, reducible, and, similarly 
to complementary series representations, determined by one real parameter A G (0, Aq), 
Ao>0. 

The following property shows the relation between the canonical representations of the 
groups 0(n, 1) and t/(n, 1) and the complementary series representations of these groups: 

Each complementary series representation is contained as a direct summand with 
multiplicity one in exactly one canonical representation, the one with the same parameter 
A. 

There is a simple relation between the canonical representations of the groups 0(n, 1)^ 
and C/(n, 1)''^ and their special representations. Namely, for A — > the norm ||/||a in the 
space H'^ of the canonical representation degenerates on a space H C of codimension 1. 
Therefore, as A ^ 0, the canonical representations tend to the identity representation 
on the quotient space H'^/H. The space H, glued to this one-dimensional subspace, has 
a positive invariant norm ||/||. In the case of the groups 0(n, 1), n > 2, this norm is 
nondegenerate; the completion 7i of the space H with respect to this norm contains the 
space of the special representation as a direct summand. In the case of the groups U{n, 1), 
this norm degenerates on some infinite-dimensional subspace Hq C H; the completion H 
of the space H/Hq with respect to this norm is the direct sum of the two special subspaces. 

The role of canonical representations in the representation theory of current groups is 
that their spherical functions 'ix{g) satisfy the infinite divisibility condition: 

*Ai+A2(y) = *Ai(y) ^XM- 

In Sees. 1-3 of this Appendix we give a general definition of a canonical state and the 
associated family of canonical representations for an arbitrary locally compact group 
G and discuss their basic properties. The subsequent sections are devoted to the canonical 
representations of the groups 0{n, 1) and ?7(n, 1). 

For the classical simple Lie groups, other approaches to the notion of canonical rep- 
resentation, various generalizations of this notion, and the related harmonic analysis are 
presented in a series of papers. See, e.g., [46, 48] and the bibliography in [48]. 

7.1. General definition of canonical states and canonical representations. Let 

G be a locally compact, noncompact topological group, let K be a subgroup of G, and 
consider the homogeneous space X = K\G of the group G, on which G acts by right 
shifts: X — > xg. We assume that on X there exists a G-invariant measure dfi{x). 

Denote by L(X) the subspace of compactly supported continuous functions on X. We 
will define elements from L{X) by their liftings to the group G, i.e., as functions f{g) 
satisfying the relation f{kg) = f{g) for A; € i^, € G. 
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Definition 1. We say that a continuous positive function (f{g) on G is positive definite 
onX if 

1) ip{g-^)^ip{g), 

2) (f{kigk2) = (f{g) for ki,k2 e K, 

3) for every nonzero function / € L{X), 

Wff - f ^{gig2^)f{gi)J{m)dii{xi)dii{x2) > 0. (7.1) 

JXxX 

The expression in the right-hand side of (7.1) is weh defined, because the integrand 
function can be projected from G x G to X x X. 

Remark. By property 2), the function (f{g) is projected to X by the map G — > K\G. 

Definition 2. A canonical state on the group G with respect to the subgroup is a family 
of positive continuous functions '^~^{g) = {'^{g))~^ on G depending on a real positive 
parameter A (degree) and satisfying the conditions 

1) ^ ~^{g) is positive definite on X in the sense of Definition 1 for small values of A > 0, 

2) for the function ^'^^■)^ = —\og'^{g), the following integral diverges: 

^ log^ {g)dfi{x). (7.2) 



L 



'X 

Proposition 7.1. Let B{xi, X2) he a continuous positive function on X x X satisfying the 
following conditions l)-4); 

1) symmetry: B(xi,X2) — B{x2,xi), 

2) G-invariance: 

B{xig,X2g) ^ B{xi,X2) for every g e G, 

3) for every nonzero function f € L{X)j for small A > 0, 

/ B~^{xi,X2) f{xi) f{x2) dij,{xi) dij,{x2) > 0, (7.3) 

JxxX 

4) the following integral diverges: 

— / \ogB{xi,X2) di2{x2) 
Jx 

(this integral does not depend on xi in view of condition 2)). 

Then ^~^(gf) = B~^{xogjXo), where xq E X is a fixed point of the subgroup K, is a 
canonical state. 

Remark. In the case when G is the group 0(n,l)orC/(n,l) and K is its maximal compact 
subgroup, there is a simple relation between the kernel B~^(xi, X2) and the Berezin kernel, 
see Section 4 below. 

Definition 3. The family of Hilbert spaces associated with a canonical state '^~'^{g) is 
the family of Hilbert spaces of functions on X with the norms 



/ ^ "^(^1^2"') f{9i) f{92) di,{xi) df,{x2), (7.4) 

JXxX 
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or, in terms of the kernels B~'^(a;i, 0^2), 

11/11^ = / B-\xuX2) f{x,)7{^dfi{xi)dfi{x2). (7.5) 

JXxX 

Definition 4. The representations of the group G associated with a canonical state 
"^~'^{g), or simply the canonical representations, are the unitary representations of G in 
the spaces by operators of the form 

{Tgf){x)=f{xg). 

Remark. If we do not require that '^~^{g) is positive definite for all A, then this represen- 
tation can be nonunitary. 

7.2. Spherical functions. Note that the norm (7.4) is defined for any compactly sup- 
ported distribution on X, and elements of the space can be interpreted as such distri- 
butions. In particular, the delta function Sxq{x) at an arbitrary point xq € X, defined by 
the formula 




^xo{x) f{xo) dflix) = /(xo). 



is an element of for every A € (0, Aq). 

Proposition 7.2. The functions Sxo{x) € have the following properties: 

1) any finite number of them are linearly independent, and they form an overdefined 
system in H^, 

2) ||4o(^)IIa = 1, 

3) for every xq, the vector Sxq{x) is cyclic in H'^, 

4) the vector 5xq{x) is invariant under the stationary subgroup of the point xq. 

Definition. Let us fix the special cyclic vector = ^xoix), where xq ^ X is a point with 
stationary subgroup K, and define the spherical function '0a(5') on G associated with the 
space by the formula 

Proposition 7.3. The spherical function can be expressed in terms of the canonical state 
by the formula 

Md) = *-^(y); 

thus it is an infinitely divisible function, i.e., 

^Xi+X2{9) = V^Ai(y)V^A2(y)- 
Indeed, setting xi = xogi, X2 = xog2, we have 

{Tg^x, 6) A = / ^~^9i92) ^xoixig) Sxo{x2) dii{xi) dfi{x2) 

J XxX 

= [ ^-\gi)6xoixig)dfi{xi)^ [ ^-\gig-^) 5xo{xi) dfi{xi) ^ ^-\g-^). 

JX JX 

The infinite divisibility of the spherical function implies the following proposition. 
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Proposition 7.4. For A = Ai + . . . + A„ there exists an isometric embedding 

commuting with the action of the group G. Namely, the map 

can he extended to a required morphism. 

Indeed, the assertion follows from the equation 

n 

valid for every g ^ G with A = Ai + . . . + A„. 

7.3. The special representations of the group G associated with the canonical 

state "^~-'^{g). As A ^ 0, the canonical state tends to one, hence the norm ||/||o = 
lim^^o ||/|Ia ^® written as ||/||o = | fx /(^) diJ,{x)\'^; the operators of the representation 
are still given by the formula 

{TJ){x) = f{xg). 

Let us study the structure of the subspace on which this norm degenerates, i.e, the invariant 
subspace H of functions satisfying the condition 



f{x) dfj,{x) = 0. 

In the space H there is a pre-norm invariant under the operators of the group G: 



'X 



2 



log *(£/i£/2 ^) f{gi) f{92) dn{xi) dii{x2). 

XxX 



Let Hq c H he the subspace on which this pre-norm degenerates. Then we can define a 
unitary representation Tg of G in the Hilbert space H obtained by completing the quotient 
H/Hq with respect to the norm 



Proposition 7.5. The representation Tg of the group G in the space H has a nontrivial 
1-cocycle a: G ^ H, namely, 

a{g)^Tg^-^, where ^ ^ S^^ix) G H \ Hq. 

Proof. First of all, note that ^ ^ H and a{g) e H, i.e., a{g) is a 1-cocycle G ^ 7i. Assume 
that this cocycle is trivial, i.e., Tg^ — ^ — Tg^o — for some ^0 ^ H. Then the vector 
^1 = ^ — ^0 is G-invariant; hence ^1 = const. On the other hand, the integral 

/ \og^{g)f{g)df,{x) 
Jx 

converges for / = ^1, since it converges for / = ^ and for / = ^o- Hence, since ^1 ^ 0, the 
integral 

/ \og^{g)dfi{x) 
Jx 

also converges, which contradicts (7.2). □ 
Corollary. The space H contains the special subspaces as direct summands. 
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Proposition 7.6. The norm of the nontrivial 1-cocycle a : G ^ H can be expressed in 
terms of the canonical state by the formula \\a{g)\\'^ = 21og^'(gf). 



Proof. Indeed, we have 



{Tg^x - Ca, ^3 Ca - ^x)x 

A=0 



d 

2 — 
dX 



A. o^*"^(^) 



(l-(?^,eA,CA))A= -2 



A=0 



A=0 



□ 



7.4. Canonical representations of the groups 0(n, 1) and C/(n, 1). Let us realize 
0(n, 1) and U{n, 1) as the groups of hnear transformations in E"'+-^ and C""*"-^ preserving, 
respectively, the bilinear form xix[ + . . . + Xnx'^ — Xn+ix'^j^^ and the Hermitian form 

xix'i + . . . + Xnx'^ — Xn+ix'^^i- In this model, elements of the groups are block matrices 
g — ^ where a and 5 are square matrices of orders n and 1, and the subgroup K 

of block-diagonal matrices ^ = ) where a E U{n) and S € U{1), is their maximal 

compact subgroup. 

Consider the homogeneous space K\U{n, 1). In the case of 0(n, 1), it is realized as the 
open unit ball D C M*^, and in the case of U{n, 1), as the open unitary unit ball D C C", 
i.e., 

D: = + ... + |2;„|2 < 1^ 

where G M in the case of 0(n, 1) and € C in the case of U (n, 1). 
In both cases the action of the group is given by the formula 

z ^ zg = {z(3 + S)~^ {za + ^) ior g = . 

Proposition 7.7. The functions 

11 - zz'*\ 

on the real and complex unitary ball, respectively, satisfy the conditions of Proposition 7.1. 
Corollary. The function 

* -^{g) = B ~^{zog: zq), where zq = 0, 

is a canonical state. 

The explicit expression for B[z, z') implies the following proposition. 
Proposition 7.8. The canonical state is given by the formula 

^-\g)^\S\-' forg^(Z f)- 
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Corollary. In matrix model b of the groups 0{n, 1) and U{n, l), the canonical states are 
given by the formula 

In particular^ 



^-^(7) = ( 1 + nL ) and *-^(t,7) = 




-A/2 



4 

for elements 7 € Z and {t, j) ^ H of the corresponding unipotent subgroups. 

The canonical representations of the groups 0{n, 1) and U{n, 1) associated with this 
canonical state act in the Hilbert spaces of functions on the real unit ball D c M and 
the complex unit unitary ball D C C", respectively, with the norms 

1 - ZZ'* -A 



f{zi)f{z2)dfi{zi)dfi{zi). (7.8) 



iD.D'{i-zz*y/Hi-z'z'y/^ 

Here djj,{z) is the invariant measure on D defined by 

dfi{z) = (1 - dz, 

where dz is the standard Lebesgue measure, A; = ^ in the case of 0(n, 1), and A; = 1 in the 
case of ?7(n, 1). The operators of these representations have the form 

{TJ){z) = f{zg). 

Passing from functions f{z) to the functions ip{z) = |1 — zz*\~^~^^'^~^^^ fi.^), we can 
realize these representations in the Hilbert spaces with the norms 



IIv'IIa ^ / \^~zz'*\ ^ (p{zi)(p{z2) dzidz2. (7.9) 
Jdxd 

In the new realization, the operators of the representation are given by 
{Tg'<p){z)-<p{zg)\zP + 6\'-'>^^^+'^ for g = (^^ 

Remark. The definition of canonical states and canonical representations can be extended 
to all groups 0{p,q), U{p,q), p ^ g ^ 1. Elements of these groups are written as block 

matrices g = , where a and 5 are square matrices of orders p and q. The analogs 

of the unit balls are the unit matrix balls, that is, the manifolds of real and complex q x p 
matrices z, respectively, satisfying the condition zz* < Cq, where Cq is the unit q-matrix. 

In the general case, the canonical state ^'"''^(^r) and the kernel B{z, z') are given by the 
formulas 

*-^(^) = |det5|-^ iovg={^^ 

B{z,z') = Idet(e,-..p| 

det^/^{eq- zz*) det^/^{eq- z'z'*) 
The case q> 1 differs from the one considered here in that the canonical state is positive 
definite only for A > q' — 1. 
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Note that the functions det^{eq — zz'*) are the Berezin kernels. In the case of U{n, 1), 
instead of the Berezin kernels we consider their absolute values. 

7.5. Relation to the complementary series representations. For each of the groups 
0(n, 1) and C/(n, 1) there exists an isometric embedding 

commuting with the action of the group, where is the space of an arbitrary comple- 
mentary series representation and iJ^ is the space of the canonical representation with the 
same parameter A. Let us describe this embedding explicitly. 

We will use the realization of complementary series representations in the space of func- 
tions f{uj) on the unit sphere 5" C in the case of 0(n, 1) and on the unitary unit sphere 
5 C in the case of C/(n, 1). 

Proposition 7.9. If is realized as the space with the norm (7.9), then for each of the 
groups 0{n, 1) and U{n, 1) the map 

r:f{u;)^^{z)^S{l-\z\')f{f^) 

(where 6{t) is the delta function onM.) is an isometric embedding ofL^ into commuting 
with the action of the corresponding group. 

Proof. For definiteness, consider the case of the group U{n, 1). The complementary series 
representation T'^ of this group acts in the Hilbert space L"^ of functions on the unitary 
sphere S = 5^*^"-^ in C" with the norm 



= / \l-ujuj'*\-^ f{uj) f{uj')dujduj', 

JsxS 

where dcu is the invariant measure on S. The operators of this representation have the form 

Tg^f{u;) = f{u;g) \u;p + 6\>^-^^ . 
Let us prove that r commutes with the action of U{n, 1). We have 
{TgTf){z) = 5{1 - \zg\^) \zP + 

Since S{1 - \zg\'^) = S{1 - \z\'^) \zp + it follows that 

{T,rf){z) = S{1 - \z\') /(g) \zp + S\'-'- = {tTJ){z). 

Let us prove that the embedding is isometric. Let ip{z) — {Tf){z) = 5(1 — 

|zp)/(|||). Passing to the spherical coordinates on D = K\U{n,l), i.e., the coordinates 
z = ruj, r > 0, uj E S, we obtain 

WTgifWl = / \1 - rr'toto'*\-^ 5{1 - r'^) 5{1 - r''^) {rr'f''-^ f{co) f{uj') dr dr' dco dcj' 
\1 - ujuj'*\-^ f{uj)J{J) du dio' = Wff. 
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'SxS 

The proposition is proved. □ 
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7.6. The structure of the Hmiting space H^', the 1-cocycles and special repre- 
sentations associated with H^. According to general definitions, the space H^, which 
is the A — > limit of the spaces of the canonical representations, contains an invariant 
subspace H C of codimension 1, namely, the space of functions satisfying the relation 

Jd 

the quotient representation in H^/H is the identity representation. Thus, as A — >^ 0, the 
representations T'^ in the spaces H'^ tend to the identity representation in the quotient 

space /H. 

Let us study the structure of the subspace H glued to H^/H. On this subspace there is 
a seminorm ||/|| invariant under the operators (Tgf)(z) — f{zg): 

ll/f = - / \ogB{z,z') f{z)J{^)d^l{z)d^l{z'). 

J DxD 

The condition f E H implies that this seminorm can be written in the form 

\\ff = -[ \og\l -zz"'\f{z)J{^di^{z)di^{z'). (7.10) 
JDxD 

Passing from functions f{z) to the functions (p{z) = |1 — zz*\~^^'^~^^\ where k = 1/2 and 
= 1 for the groups 0(n, 1) and U{n, 1), respectively, we obtain the following expression 
for the norm: 

\\(pf = -l \og\l- zz'*\^{z)(p{z')dzdz', (7.11) 

J DxD 

where dz is the Lebesgue measure on D. In the new realization, the operators of the 
representation are given by the formula 

{Tg^){z) = ^{zg) \zp + 5|-2Mn+i) for g = 

In the case of 0(n, 1), the norm (7.11) is nondegenerate; in the case of U{n, 1), it de- 
generates on some infinite-dimensional subspace Hq. Consider these cases separately. 

7.6.1. The case of the group 0{n, 1). Expanding log |1 — zz'*\ = log(l — zz'*) into a power 
series, we obtain the following expression for the norm \\(p\\'. 

|fc|>o ■ 

where, as usual, A:! = ki\, \k\ — ^ ki, and = H ^i^- follows that this norm is strictly 
positive. Denote by TC the Hilbert space obtained by completing H with respect to the 
norm \\(p\\. 

Proposition 7.10. The space 7i contains the space C of the special representation of 
0(n, 1) as a simple summand. If C is realized as the space of functions f{cv) on the unit 
sphere S"'~'^ C M", then the embedding t : C —> 7i has the form 

r: f{uj)^^{z) = 5{l-zz*)f{^), 
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where S{t) is the delta function on M. . 

The proof is the same as in Proposition 7.9. 

Proposition 7.11. For every vector^ e H'^\H, the function a (g) = Tg^ — ^ is a nontrivial 
1-cocycle G ^ 7i. If ^ = S{1 — l^;^), then a{g) G C, i.e., it is a 1-cocycle G —> C 

Corollary. The subspace 7i' = HQC does not contain nontrivial 1-cocycles. (It decomposes 
into a direct integral of principal series representations.) 

The structure of the special representation of the group 0(n, 1) was considered in § 2 of 
this paper. 

7.6.2. The case of the group U{n, 1). We have 

log |1 - zz'*\ = i log(l - zz'*) + i log(l - z'z*). 
Expanding each term of this sum into a power series, we obtain the following proposition. 
Proposition 7.12. The norm ||£/9|| on H can he written as the sum 

II Il2 1 II ||2 , 1 II ||2 

\m\ ^ ^\m\+ + ^\m-^ 

where 

fc>o ■ fc>o ■ 

These norms, as well as the original norm \\^\\, are invariant under the action ofU{n, 1). 

It follows from the explicit expression for \\ip\\ that this norm is positive; however, in con- 
trast to the case of the group 0(n, 1), it degenerates on some infinite-dimensional subspace 
Hq. Let TC be the completion of the quotient space H/Hq with respect to this norm. 

The invariance of the norms \\(p\\± implies the following proposition. 

Proposition 7.13. The space 7i is the direct sum of two invariant subspaces: 

H = H+®H-, 

where n+^{ipGn\ \\ip\\- ^ 0}, H- = {(f e H \ \\ip\\+ = 0}. 

The representations of the group U{n, 1) in these subspaces are irreducible, and the 
projections of every nontrivial 1-cocycle U{n, 1) H to these subspaces do not vanish. 
Thus the representations of C/(n, 1) in the spaces H± are special. 

The structure of these two special representations of the group U{n, 1) was considered 
in § 6 of the paper. 

§ 8. Appendix 2. Derivation of some formulas 

8.1. The Fourier transform of the functions (a^ + |7p)~^/^ and |7|~'*^ in R"^"^, 
where I7I = (E^=l^7?)^^^ is given by 



/ 

Jr 



^ -1 



(«2 ^ |^|2)-A/2^^(^,,) ^ \^\-^ K^{a\C\), (8.1) 

2 2-ir(A/2) 2 
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/ ,H(.-,} - 1 - A)/2) ^^^,_„^i^ ^^2^ 

where the coefficient c„, the same in (8.1) and (8.2), depends only on n. 
The Bessel function Kp is given by the formula (see [51, Vol. 2]) 

where 

oo 2m+p 

W = E „!r(L + p + i) - 

m=0 ^ < ' J 

Remark. For integer values of p, the series for Kp contains terms with log z; for half-integer 
values of p, the expression for Kp can be reduced to a simpler form: 



(n + k) 



^ A;!(n- A;)!(2^)^" 

Let us derive formulas (8.1) and (8.2) for n > 2. Passing to spherical coordinates, we 
can bring the first integral to the form 

roo rip 

Ji = c / (a2 + r2)-^/2e^l^l^°^'^r"-2 sin"-V(^<P(^r. 
Jo Jo 

Integrating with respect to we obtain (see [53, formula 3.915.5]) 

-3 -1 

Ji = c|^|-V / (a^ + r2)-VVV J^(|^|r)dr, 
Jo ^ 

where Jp(7) is a Bessel function of the first kind (see [51, Vol. 2]). Analogously, for the 
integral (8.2) we obtain the expression 



/•oo 

n — 3 / n — 1 



Integrating with respect to r, we obtain for Ji and J2, respectively, formulas (8.1) and 
(8.2), see [53, formulas 6.565.4 and 6.561.14]. 

Formula (8.1) immediately implies its complex analog: 

8.2. Formulas for integrals over the unitary unit sphere S C C^. The following 
formula holds: 



In particular, 

n 



a;Y^a^- .in^'"^ ' e Z!f , (8.6) 

' {\k\+n-l)\ +' ^ ^ 



where o;^ = 11^^% = 11^*!, 1^1 = E^i- 



74 

Proof. Consider the following integral over the interior of the unit ball in C"^: 



» n 

1= / X{\zirdti{z), 



where dfi{z) is the Lebesgue measure on C". Passing to the spherical coordinates z = rco, 
r > 0, a; € (S, we obtain 



Jo ^ ^ 2a Tai + n) ^ ^ 



On the other hand, passing to the polar coordinates zj — rje'^'^^ , j — 1, . . . ,n, we obtain 



P n „ n 



niLiHf + 1) 



= TT 



r(i^a, + n + l) 

Comparing these expressions for /, we obtain (8.5). □ 

8.3. The multidimensional analog of the Dougall formula. For every n > 1, 

-1 

Yi (n ^(^^ + + 1)) 

1^1=0 i=l (g_7) 
_ ^(^(a^+&^) + l) 

~ r(E a, + 1) r(E &z + 1) U7=i + &i + 1) ' 

where |^| — h + . . . + In, k € ^ , or, in a nonsymmetric form, 

n—l ^ 

J2 (r(-|r| + a, + i)r(r/| + 6, + i) l[r{k + ai + i)r{-k + bi + i)y 

«'ez"-i ^=l (8.8) 

_ ^(E(a^+6^) + l) 

" r(E a, + 1) r(E 6, + 1) niLi + 6^ + 1)' 

where 1^1 = ^=1'^^ 

Proof. For n = 2, formula (8.8) coincides with the classical Dougall formula, see [51]: 

J2 (r(^ + ai + i) r{i + a2 + 1) r(-/ + A + 1) r(-/ + /32 + 1)) ~ 

/=— oo 

r(ai + a2 + Pi+p2 + l) 



r(ai + A + 1) r(ai + /52 + 1) r(a2 + /5i + 1) r(a2 + /53 + 1) 
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Let us prove (8.8) for every n > 2 by induction on n. By the Dougall formula, 

OO 

J2 + an + 1) m'\ +bn + l) r{ln-l + On-l + 1) r(-/,_i + bn-1 + 1)) 

ln-l = — 00 

_ r(a„_i + h,>-i + an+ + 1) 

~ V{an-1 + hn-l + 1) r(an + hn + l) + an-l + an + 1) r(|/"| + hn-1 + &n + 1) ' 

where I" — li + . . . + ln-2- Substituting this equation into (8.8) yields 

^ _ r(an-l + hn-l + an + hn+l) j 

T{an-i + bn-i + l)T{an + bn + l) 

where 

n-2 _^ 

/n-1 = ^'(r(-|r|+an-l+an+l)r(|r|+6„_i+&n+l) [] ^(^^+«i + l)^(-^^ + &^ + l)) i 

i=l 

the prime means that the sum ranges only over li, . . . , ln-2- By the induction hypothesis, 
, r(E(a. + &.) + !) 

J-JT 2^ (-) • 

r{an-l + bn-1 +an + bn + l)r(^ tti + l)r(^ h + 1) [I.t" 1 ^i^i + h + 1) 
This implies (8.8). □ 
Remark. The following generalization of (8.7) is obvious: 



" -1 

J2 (n ^(^^ + + r(-^z + bi + 1)) 



r{^{ai + bi) + l) 

" T{J2ai + m + l)T{J2bi-m + l) 11111^(0^ + 6^ + 1) 

for every m E Tj . 

8.4. Calculation of the norm of the vector fp. The norm of the vector fp in the space 
L^{p) equals 



Proof. According to the original definition. 



where 

Q{t) ^t-^ e^/^ Kx^{t/2). 
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8.4.1. First let us reduce this expression for the norm to 
where c(A) = 

For this, make the change of variables u = p~^/'^ u', v = p~^^'^ v', u' = z + v' and use 
the identity 

— 1/2/* * *\ /* //* * 

p ' [vu — uu — vv ) — —zv — V V — zz . 
In the new variables z and v', v' ^ v, we obtain 

Wfpf = c(A) p-l^l [ {Z + v)PvP Q(|2|2) e -i^v*+vv*+zz*) 

l\{p-l)\ Jc»-ixC"-i 

Let us substitute into this formula the power expansion e~^^* = X^^g^""^ ^ fcl' ' 
Throwing off the terms that integrate to 0, we obtain 

The first integral in the obtained equation equals p\. The second one can be reduced to 
the form 

1= f s^Q{Esi)e-^'^ T\dsi 

„ n— 1 n—1 n—1 

/ {s-y2s^y'{l\s,)Q{s)e-'ds r\dsi 

'"^+ 1=2 i=2 i=2 

fOO 

^0 



+ n-2)! 



Thus 



8.4.2. Now let us check that substituting the expression for Q{t) into (8.11) yields 

- (8-12) 



^" r(A/2)^ '^-^ ^^|(^_^),r(|/|+n-|) '• sin7r(n-A) 



X (r(|/| + n--)r(-|/|-n + A + i) []r(/, + i)r(p,-/, + i))' 

zez"-i z=i 
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Let us use the formula 



Jo (2a)Mr(/^ + l/2) ' 

see [52, Vol. 1, p. 331, (28)]. For u = /i = |/| + n - a = 1/2, we have 



„i/2 r(|/| + n-l)r(|/|+n-A) 



r(|/| + n-|) 

This implies (8.12). 



8.4.3. Finally, apply the Dougall formula (8.8) to the sum (8.12). In our case, = 0, 

A 

2 



hi — Pi for i ^ n, ttn — —n + A, and a — n — ^ — 1, bi — pi. Therefore 



2 



C(A) TT 



3/2j^l^lM! £M 



A 



sin7r(n - A) r(f ) r(|p| + n - f ) r(A - n + 1) p ! ' 
Substituting the equality sin7r(n — A) r(A — n + 1) — tt r~^(n — A) yields (8.10). □ 

8.5. Derivation of formula (6.46) for the operator Tg in the Bargmann model: 

rp+(^a JcN _ (_-,^\k\ -a+\k\ k 

Proof. It follows from Proposition 6.14 that 

where, according to (6.45), the function il^i{p, z) is given by the formula 

oo oo 
-oo 

oo 2 

= (-l)l'=l r(a + n) p-^+i ( y e-P^ (-C)"" (-C)"'^'+" dt) z\ C = ^^ - 

— oo 

According to (6.44), the equality f{p)z^ = Til^i{p,z) implies that the function f{p) is 
given by the formula 

f{p) = P-—— / ^,[p,z)-z^e-p\^\ df,{z). 

Let us proceed to the calculation of this function. Taking into account that —pC,—p\z\'^ = p(, 
we have 



f{p) = + pl^l r [ (-C)-" (_^)-l'^l+" dp{z) dt. 



oo JC 



n-1 
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Making the change of variables t ^ p ^t, z ^ p ^/"^z, we obtain 



where 



f{p) = c{a, k) p 



c{a, k)^ / (-C)-" dli{z) dt. 



To calculate c{a,k), replace the complex variables Zi by the variables = ^ and (pi = 
argz^. Since the integrand does not depend on (pi, we obtain, setting r = ^r^, 



/OO /• 
/ e-(^+^*)(r + zt)-^(r-zt)-l^l+'^ TTrf^dr.dt 

poo 
^0 



^-oo ^0 

Now, setting r + it = ue^^^ w>0, — f < (p < f, we bring the integral to the form 



2 2- 

"OO 



c{a, k)=c' / e-^^^'' g-(-|fc|+a+n)<p cosl^l+"-V^^/^(^i) ^^V^- 
J-l Jo 

Integrating with respect to u yields 

c(a, /e) = c' r(a) / ' e -Hk\+2a+n)^ cosl^l+^-2 



r(o; + n)r(-a+ |/e|)' 

see [53, p. 386, formula 9, expression for the last integral]. 
As a result we have 

Tsip- Z^) = C (-1)1^1 -JM^^-a+|fc| c > 0. 

i {—a + \k\) 

Since T^^ = id, it follows that c = 1. □ 
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